Subject: [rrg] Purpose and mechanisms of a Core-Edge Separation scheme - Ivip in particular

From: Robin Whittle <[email-masked]>
Date: Thu, 28 Jan 2010 00:11:10 +1100

This is a walk-through my understanding of what a CES architecture
aims to do and how it would work.  I am using Ivip as an example, and
I am including global mobility in addition to the basic goals of
scalable routing.

This stuff can be hard to understand.  I hope by reading about it in
various ways that people will understand it better.

  - Robin

In msg05799, Patrick Frejborg wrote:

> In a nutshell, CES is all about taking out the multi-homed PI
> addresses from the DFZ - transforming them into PA-addresses
> (RLOCs, which are aggregated) and provide a multi-homing solution
> for IPv6 so it will not further expand the size of DFZ.

It is not just about transforming an actual address or prefix of
addresses from PI use to a new use - SPI (Scalable PI) for Ivip, or
EID for LISP.  It is certainly not about transforming actual
addresses or prefixes of addresses from PI to PA.

It is broadly speaking about generating a growing subset of the
address space which are "edge" addresses - SPI space for Ivip - which
is suitable for end-user network use because it can be used for
portability, multihoming and inbound TE, in a manner which scales
well: does not impact the DFZ control plane, or the DFZ routers'
FIBs, anywhere near as much has the current only way of doing it - PI
space advertised in the DFZ.

Then more and more end-user networks can adopt this type of space,
gaining the benefits, in a scalable fashion.  Some will be new
networks.  Some will have used PA space in the past.  Their SPI
addresses will be different from their PA address.  (Though they may
continue to use their small piece of PA space to run an ETR, through
which they can use their SPI space - and maybe get another PA address
from another ISP so they can multihome, with an second ETR on that
address too.)

Others will have PI space now. They could either convert their PI
space into SPI space, or return it to their RIR and hire some SPI
space from a company which provides this.

Because (with Ivip) SPI space can be split up into micronets as small
as a single IPv4 address, or an IPv6 /64, this kind of space can be
used much more efficiently than with the current approach, in which
prefixes of 256 IPv4 addresses, or 512, 1024 etc. must be used.

Here is a fuller description.  The first part is a statement of my
understanding of the goals of scalable routing, extended to include
support for global mobility.

Mobility is not a formal part of the RRG's goals, but we obviously
need to do it.  In the future, I am sure, most Internet hosts are
going to be wireless mobile hand-held devices.  I think global
mobility - a single IPv4 or IPv6 address or range of addresses, no
matter what the point of attachment, including behind NAT, is the
best and most practical approach to mobility.  The TTR Mobility
architecture provides this - and as far as I know this is the only
way to achieve global mobility.

The second part is an Ivip-specific description of how a CES works -
not counting some important details such as PMTUD discovery
management if encapsulation tunneling is used, or the adoption
instead of tunneling with Modified Header Forwarding - either
initially or in the long-term.

Purpose 1 - general scalable routing and addressing
---------------------------------------------------

There are, or will be, probably 10 million (10^7) or so non-mobile
end-user networks which want or need portability, multihoming and/or
inbound Traffic Engineering (TE).

Brian Carpenter and I have independently suggested this figure in the
past and recently.  I plan to write more about this.

We can't satisfy this want/need with current techniques because the
only way of doing it would be to have each such end-user network
advertise at least one PI prefix in the DFZ.

With IPv4, it could be tricky to find enough such prefixes, unless
they were longer (less addresses) than /24 - which is not inconceivable.

Many such networks lack the resources (financial and technical) to
obtain and manage PI prefixes like this.

However, the most important problem is that doing this would
overburden the DFZ routers.  The overall DFZ control plane would have
to handle 10 million or more prefixes, compared to the current 300k.
 The route processors, with their RIB data structures, would need to
handle all this - and the RIB & route processor load scales in rough
proportion to the number of neighbours each DFZ router has, because
each neighbour engages in a continual two-way parley about every
advertised prefix.  (Not counting a few which might be aggregated
into one.) Also, each DFZ router would need this number of prefixes
in its FIB too.

The aim is to enable the wants and needs of these end-user networks,
regarding portability, multihoming an TE, within their resource
constraints, and while not excessively burdening the DFZ.

Ideally, existing PI users could migrate to the new system and so
reduce the number of prefixes in the DFZ.

Ideally (though this is not in the RRG goals) the new arrangement
would enable more efficient use of IPv4 space.

While the support for these 10M or so end-user networks will have
some impact on the DFZ control plane, RIBs and FIBs, it is vital that
the impact on average per end-user network is very much less than at
present with PI prefixes.

Purpose 2 - supporting mobility on a massive scale
--------------------------------------------------

In the future there may be 10 billion (10^10) wireless-linked mobile
devices, such as cell-phones, laptops or whatever.

Whatever changes we make to the routing and addressing system should
also support each such device working well with the Internet - though
for the larger numbers this probably means the IPv6 Internet rather
than the IPv4 Internet.

This is not a formal goal of the RRG, but I think many people agree
we should make a single set of architectural changes to solve both
challenges.

It so happens that a CES architecture which works well for non-mobile
end-user networks can also support the TTR Mobility architecture.
The Translating Tunnel Routers appear to be ordinary ETRs and ITRs to
the CES system.  MNs (Mobile Nodes) make two-way tunnels to one or
more (typically) nearby TTRs - and these handle the MN's incoming and
outgoing packets for its globally portable, micronet of SPI (Scalable
PI) address space.  Mapping changes are not needed every time the MN
gets a new access-network address - they are only desirable when the
MN moves a long distance, such as 1000km or more.

The other approach to making CES support mobility is to have the MN
be its own ETR and probably ITR.  If so, the MN can't be behind NAT
and each such MN needs a global unicast address of its own to be its
own ETR.  This approach also requires effectively instant changes in
ITR behavior to maintain connectivity in the frequent instances where
the MN gets a new access network address.  No CES architecture can do
this.  TTR Mobility is a much better approach.

Assuming the CES mapping system can handle 10^10 micronets, rather
than the 10^7 needed without ubiquitous mobility, and assuming the
CES network elements (ITRs and ETRs) can handle the extra traffic for
all these mobile devices, then the CES scheme needs no elaboration to
support the TTR Mobility architecture.  A CES architecture with
real-time mapping (Ivip) would be better than LISP (non-real-time) -
but either would work.

So it is possible and desirable to plan a CES architecture to achieve
both goals.

How a CEE architecture might attempt to achieve these goals
-----------------------------------------------------------

Core-Edge Elimination involves creating a new namespace for uniquely
identifying hosts.  Applications request the IP stack to set up
communication with other hosts which are specified by their
Identifier address.  (Some people object to the use of "address" in
this context.)

The stack handles the hosts acquiring and relinquishing their
"physical" addresses - the IP address by which they actually send and
receive packets - to retain session continuity during these changes,
and in the long-term to provide complete portability of the hosts
Identifier "address" no matter which one or more ISPs they are using
to connect to the Net.

Therefore, the new stack functions, and the applications which work
entirely in terms of Identifiers, enable hosts (and whole networks of
hosts) to have portability, multihoming and inbound TE.  Mobility is
basically a fast form of multihoming - but it involves the MN
acquiring addresses on all sorts of access networks, on an ad-hoc
basis.  If the CEE architecture can handle mobility (not all attempt
to do so) then it will need some fancy authentication arrangements so
the session can continue with packets being sent to and from some new
access network physical address than was previously used.

Because applications only use Identifier addresses, the hosts
themselves can use any kind of access network address (though
probably not behind NAT) - any kind of "Locator" or "physical"
address.  If all hosts and all their applications follow the CEE
paradigm, then there is no need for PI addresses at all.  ISPs will
each have a handful of prefixes they advertise in the DFZ, and all
end-user networks will be temporarily be given an address or prefix
from these.  This is PA address space, and it will be fine for all
hosts, since all applications select their correspondent hosts
entirely by using Identifier "addresses".

In CEE, there is no separation of a subset of the "core" addresses to
become "edge" addresses - which is what happens in a CES architecture.

In a CEE architecture, all the addresses the hosts physically use are
 ordinary IP addresses (or some new construct which replaces IP
addresses).  All hosts physically connect using a single type of
Locator address, and there is no distinction between subsets of this
space to be reserved for "edge" networks, such as those end-user
networks which want portability, multihoming and TE on a scalable
basis.  All hosts use a PA address from each one or more upstream ISP
- so there's no need for any end-user network to have is own PI
assignments of Locator address space.

There are now two namespaces: Locator and Identifier.  There is no
core and edge distinction between different classes of Locator
address - they are all PA addresses from the point of view of
end-user networks.  Identifiers are from a separate namespace, which
is global - and each Identifier is unique in the world.  Identifiers
have nothing to do with ISPs and have no relationship with geography
or network topology.  (Some CEE architectures allow non-unique
Identifiers, but this look like a lot of trouble to me.)

The first trouble with CEE is that it involves changes to all host
stacks, APIs and applications - in order to replace the current
methods by which hosts communicate, with benefits only arising when
two CEE hosts, with CEE applications, communicate.  The second is
that it can be argued that it is undesirable to expect all hosts to
take on more responsibility for routing and addressing - because some
or many of them are on slow and potentially unreliable links.  See:

 http://www.ietf.org/mail-archive/web/rrg/current/msg05745.html
 http://www.firstpr.com.au/ip/ivip/RRG-2009/host-responsibilities/.

How a CES architecture - Ivip - achieves these goals
----------------------------------------------------

Some of the following applies to LISP, though with different terminology.

Ivip creates a new subset of the global unicast address space.
Addresses previously used for conventional purposes are now in this
subset - and the subset grows as more people who control address
space choose to use it in the new way.

Currently we have ISP prefixes, which when given temporarily, or
semi-permanently, to the ISP's customers (usually a small section of
the actual prefix the ISP advertises) is known as a PA prefix.  This
is "Provider Assigned" and the end-user network only gets this space
when using this ISP.  PA space is not portable, or suitable for
multihoming or TE.

We also have PI prefixes - those which belong to some larger end-user
networks, and are the cause of the routing scaling problem.

Most home, SOHO and other small business and organisational end-users
are doing fine with their current PA arrangements, which are
acceptably reliable.  Most of these end-user networks don't need
portability, multihoming or TE.

For the subset of end-user networks which do need these things (10^7
or so) and for the 10^10 mobile end-user networks of the future, Ivip
creates a new kind of address space - a subset of the existing global
unicast address space.

This is neither PI or PA.  It is "Scalable PI" address space - SPI.

To do this, multiple BGP-advertised prefixes known as MABs (Mapped
Address Blocks) are used.  (LISP has the same thing, but there is no
LISP term for them.)

For instance, an IPv4 MAB might be 12.34.0.0 /16.  This is advertised
in the DFZ by multiple DITRs (Default ITRs in the DFZ). DITRs are
arranged around the Net and work like any other ITR - however a given
DITR may only advertise a subset of all the MABs in the entire Ivip
system.  Overall, all MABs are covered by widely distributed DITRs.

For more explanation of the commercial arrangements for this and for
the mapping system, see:

  http://tools.ietf.org/html/draft-whittle-ivip-arch
  http://psg.com/lists/rrg/2008/msg01158.html

It is possible for an end-user network which already has PI space to
convert some or all of this into one or more MABs.  In this case,
they are the sole users of each MAB - but they could hire out some of
the space.  This does not make them ISPs, because having MAB space
does not give Internet connectivity.

The typical arrangement will be for multiple companies to acquire
MABs and hire out the space within them, in small chunks, to large
numbers of end-user networks which want portability, multihoming, TE
and/or mobility.

This section does not concern mobility - but there is a Mobility
section at the end.

The following description concentrates on end-user networks which
rent their SPI space from a MAB company - a single range of SPI
addresses in a single MAB.  However, an end-user network could rent
space in multiple MABs from the same company or from multiple companies.

These end-user networks may never have had any address space before -
so they are new networks.  They may previously have had PI space, but
have abandoned it.  If they had PA space, such as the single IPv4
address they get with their DSL etc. service from their one ISP, then
they will presumably retain that PA space, or whatever PA address
their ISP gives them.

Renting SPI space on its own is of no use.  The end-user network
needs to pay at least one ISP for Internet access, and it needs an
ETR.  It also needs to either control the mapping of its space
directly, or pay some specialised company to do it for them.

The end-user network gets a range of SPI addresses - any integer
number.  For instance, end-user network AAA gets a User Address Block
(UAB) of 6 IP addresses:

   12.34.5.60
   12.34.5.61
   12.34.5.62
   12.34.5.63
   12.34.5.64
   12.34.5.65

(For IPv6, the units by which address space can be split up into
micronets is a /64 prefix, so the equivalent would be six contiguous
/64 prefixes.)

AAA can decide to use the whole UAB as a single micronet.  A micronet
is a integer number of IPv4 addresses (or IPv6 /64s) in a single MAB,
which are all covered by the same mapping.

In Ivip, the mapping is a single ETR address.  There is no need to
give ITRs choice between ETRs, because if the currently mapped ETR
should not be used, the end-user network will (directly or
indirectly) cause the mapping to be changed to point to another ETR
which should now be used.  Ivip does this in real-time - a few
seconds at most.  (Less than a second, globally, looks technically
possible but it is best to think of 2 to 3 seconds.)

AAA can divide this 6 address UAB into micronets however it likes -
including into 6 micronets of a single IPv4 address each.  Then it
could map each micronet to any ETR in the world.

With a single link to an ISP, such as with its existing DSL link, AAA
gets a single PA IPv4 address 9.9.9.9.  It can still use this however
it likes, but if it wants to use some of its SPI space via this link
to the Net, then it needs to create an ETR at this address.  The
router or whatever (router function in a DSL modem, or an a server
running *nix) needs to accept IP-in-IP packets and decapsulate them.

Ivip ITRs encapsulate packets with the outer header's source address
being that of the sending host.  If the packet arrives from the DFZ
and passes through the ISPs BR filtering system (which may be set up
to reject any packet with a source address matching any of the ISP's
prefixes) then it will arrive at the ETR function in AAA's router.

The ETR function strips off the outer header and compares the inner
source address with that of the outer header's source address.  If
the two match, the packet is then ready to be forwarded.  If not,
then the packet is dropped, because it is apparently an attempt by an
attacker to get the ETR to forward a packet with a forged source
address.  (For the purposes of this discussion, I assume the ISP has
its own ITRs so it never sends packets out to the DFZ with SPI source
addresses - otherwise, this BR filtering by source address would
prevent hosts in its own networks being able to successfully send
packets to AAA's micronet address.)

If the packet is not addressed to one of AAA's SPI addresses, then it
 should be dropped.  (This could result from an attack or from some
other SPI-using network erroneously mapping their micronet to this
9.9.9.9 ETR address.)

So now the router has the packet, addressed to one of the SPI
addresses, and can do whatever AAA wants with it.

AAA could have all 6 addresses as a single micronet.  Then it could
have 6 hosts, one for each SPI address, at its site.  AAA can still
use its 9.9.9.9 address as before - typically it would have been
used, and may still be used, as the public address of a NAT function,
with a bunch of hosts behind NAT on the LAN.

AAA might use one of the SPI addresses for a NAT box, another for a
web server, another for a mail server or whatever.

AAA could split the space into multiple micronets and map each
micronet to any ETR address in the world.  It could map all the
micronets to 9.9.9.9, which would have the same effect as having all
6 IP addresses in one micronet, and mapping that to 9.9.9.9.

(It is not possible to map the micronets to any address in the SPI
subset, but any other global unicast address will do.  It is AAA's
responsibility to only map its micronets to ETR addresses which it is
using.)

AAA might have a staff member who works from home, and needs for some
reason to have a portable, public address.  So they might have a
single IPv4 address micronet mapped to their DSL home address.

(For simplicity, I am assuming these DSL addresses are fixed or at
least highly stable - which is not necessarily the case in practice.)

AAA might set up branch offices and want each one to get a single
stable, public, global unicast IPv4 address - no matter how those
offices choose different ISPs.  So it ensures each of those offices
has a stable address DSL service or whatever, and maps a micronet to
each one - each office needs to run an ETR function at its DSL
address, as just described for AAA's main office.

These offices can be anywhere in the world, with any ISP.

Now back to AAA's main office.  AAA can change its ISP so its DSL or
whatever service arrives with a single IPv4 address 7.7.7.7.  So it
changes the mapping of whatever micronets it was using in the main
office from the previous ETR address of 9.9.9.9 to 7.7.7.7.

Let's say AAA wanted to multihome its main office - and for
simplicity of explanation, that the main office only needed a single
IPv4 address.

AAA keeps its DSL service with 9.9.9.9 and gets an HFC service with
an address of 8.8.8.8.

Now it sets up an ETR function for each link, on each address.  It
can send out packets on either link (more on outgoing packets below)
and it can receive packets addressed to its SPI addresses via tunnels
from ITRs to the ETR function on either link.

Let's say the DSL link was superior due to lower latency and/or
better upstream speed.  I will assume the single micronet containing
just one SPI address:

  12.34.5.60

is all that is required for this main office.

AAA sets the mapping for this micronet to 9.9.9.9 and all the ITRs in
the world will tunnel packets addressed to 12.34.5.60 to the ETR
function which attaches to the DSL link.

If the DSL link fails, AAA needs to somehow (ideally quickly and
automatically) change the mapping of this micronet to 8.8.8.8.

Maybe AAA could do this manually once either its DSL ISP, or its DSL
link, dies.  It could use wireless Internet to log into the company
which it uses to send mapping commands, or maybe it could phone them
and give a username and password etc. with the new address to use:
8.8.8.8.

In general, networks like AAA will want to pay a separate company - a
Multihoming Monitoring company (MM) to do this for them.

The MM company has a bunch of servers all over the world and fancy
software so several of these servers continually probe the
reachability of 12.34.5.60 via both ETRs.  So the router at AAA's
office needs to have some UDP port, or some other mechanism, by which
it can send out replies to these probe packets.  Ping would be
sufficient.

AAA tells MM how often it wants these probe packets to be sent, and
what algorithm to use to decide when to change the mapping to 8.8.8.8
- and likewise what algorithm to use when deciding when to change it
back to 9.9.9.9 once the DSL-link ETR is working properly again.

AAA gives MM the credentials it needs to control the mapping of the
12.34.5.60 micronet.  MM's servers do this via some SSL system, HTTPS
or whatever, with the Update Authorisation Server which AAA uses for
changing mapping.

Now, if the DSL ISP, or the DSL link, fails, within a few seconds
(depending on how frequently the MM servers probe reachability) the
MM system will decide to change the mapping to 8.8.8.8 - and so
connectivity will be restored.

These probe packets look like any other packet which has been
encapsulated by an ITR - they are IP-in-IP packets.  However, the MM
servers generate these directly - they don't need an ITR.

The outer and inner source addresses would be the address of
whichever MM company server sent the probe packet.  The inner
destination address will be 12.34.5.60.  The outer destination
address will be 9.9.9.9 if the MM server is probing via the DSL ISP,
 link and ETR - or 8.8.8.8 if it is probing via the HFC ISP, link and
ETR.   Typically the MM company would send these probes from multiple
sites near and far from these ISPs

The response to the probe packet should go out the link the probe
arrived on.  Obviously if a probe arrived via DSL link, to the
9.9.9.9 address and the ETR function at this address decapsulated and
forwarded it to whatever handles the 12.34.5.60 SPI packets - and
then the resulting response was sent out the HFC link, then failure
of the HFC link would make it look like the DSL link had died.

In this case, AAA is running its own ETRs.  The ETR functions are
probably in the same router that handles the whole network, so there
is no concept of an ETR being disconnected from the destination network.

An alternative, and better scaling, arrangement is for an ISP1 to
have an ETR on one of its addresses, such as 9.9.99.99, with links
(tunnels, direct connections via some physical means or whatever) to
multiple end-user networks.

Then AAA would receive, via its DSL service, packets from this ETR.
The ETR could serve the needs of many end-user networks, large and
small.  This is more efficient, since a single IPv4 address for a
single ETR serves the needs of many end-user networks.  (This address
efficiency doesn't matter with IPv6.)

In this case, the probing by the MM company proceeds just as before.
 The MM company is not probing reachability of the ETR - it is
probing reachability of the end-user network *via* the ETR.  The ETR
decapsulates the packet, sends it to the router at AAA's office, and
the router sends back a response packet, out the DSL link.

If the DSL link dies, then the MM servers will receive no response
packets.  After a few seconds or non-response (as chosen by AAA) the
MM system will change the mapping of the micronet to that of the DSL
link ETR.  This takes a few seconds at most to get to all the ITRs
which are tunneling packets addressed to this micronet.  So
multihoming service restoration can be achieved in a few seconds.

Note that with Ivip, the probing and the decisions about how to
change mapping due to multihoming failures, is entirely outside the
Ivip CES.  With LISP and other CES systems, the probing (or however
failures are detected) must be done by the ITRs individually, and
each one must make its own decision about choosing another ETR from
the list of ETRs supplied in the more complex mapping information.

Ivip ITRs have no load-sharing functionality.  (LISP ITRs do.) To do
basic load sharing (the most common form of inbound TE), here is how
AAA would organise its network.

Firstly, the incoming traffic would need to be split in some way over
at least two separate IP addresses.  Some of the incoming packets
need to be addressed to one or more SPI addresses in one micronet X
and the rest need to be for one or more SPI addresses in at least a
second micronet Y.  There may be three or more such micronets, each
with one or more IPv4 addresses (or IPv6 /64s) receiving incoming
packets.   In this example, I assume the incoming traffic is roughly
evenly split over two single SPI address micronets:

   12.34.5.60  X
   12.34.5.61  Y

This could be done in several ways.  Even if to the outside world
there was a single FQDN which was the destination of all this
traffic, DNS could return both these addresses, so on average the
sending hosts would be sending half the packets to the X micronet and
the other half to the Y micronet.

AAA configures its service with the MM company to probe reachability
to hosts on both micronets, via both ETR addresses.  The
decision-making algorithm is configured to achieve:

  If the network is reachable via both ETRs, then:
      Map micronet X to the DSL ETR address:   12.34.5.60 >> 9.9.9.9
      Map micronet Y to the HFC ETR address:   12.34.5.61 >> 8.8.8.8

  If the DSL ETR is working and the HFC one is not:
      Map micronet X to the DSL ETR address:   12.34.5.60 >> 9.9.9.9
      Map micronet Y to the DSL ETR address:   12.34.5.61 >> 9.9.9.9

  If the HFC ETR is working and the DSL one is not:
      Map micronet X to the HFC ETR address:   12.34.5.60 >> 8.8.8.8
      Map micronet Y to the HFC ETR address:   12.34.5.61 >> 8.8.8.8

  If the network is not reachable via either ETR:
      Send an SMS message to the cellphone of AAA's IT manager!

AAA would also configure its service with the MM company so the
mapping would be returned to the normal state (the first of the four
above) for instance 2 minutes after both ETRs were found to be
operating properly again.

In this example, simple load sharing is achieved, and it can easily
be seen how it could expand to multiple other links.  Each micronet
could have more than one IPv4 address.

The same approach could be used if the HFC service had fluctuating
downstream capacity (which is quite likely) due to the shared nature
of its downstream RF channel - other customers in the area could
demand more and so AAA gets less.  The HFC link might be usable for a
mailserver, but the DSL link is more suitable for HTTP to the web
server at the office.  So this is not so much load sharing, but
splitting traffic over two links for other reasons - which is still
inbound TE.  In this case the first micronet X would be for the one
or more SPI addresses which were used for the web server.  The second
micronet Y would be for the SPI address used by the mailserver.

Let's say AAA had multiple streams of packets coming in, with varying
data rates at different times.  Maybe it receives website traffic in
response to a TV program which screens at different times in
different countries or parts of the country.  Maybe its servers
handles multiple quite different bodies of traffic for purposes which
rise and fall in popularity hour-by-hour, according to all sorts of
unrelated variables.

If it can split the destination SPI addresses for these streams into
different micronets, such as 4, 6 or more, then depending on the
traffic arriving on each at any given point in time, it can map the
micronets between its two or more ETRs, and so dynamically balance
the flows to maximise the utilisation of its links, while reducing
the chance of any one link being congested.

While each Ivip mapping change would incur a fee (I guess 20 cents to
a fraction of a cent), it may still be highly attractive to use this
at peak times, if it meant that it was possible to dynamically
balance these loads over several less expensive links, rather than
paying perhaps a great deal more for a faster link.

In this case, assuming AAA is still using MM to control the mapping
of its micronets, it needs to work out with MM a way its router,
and/or its IT managers, can tell the MM system which micronet should
be mapped to which ETR.  The MM system will pass on these requests,
by changing the mapping, directly - as long as all ETRs are working.

None of these MM company activities absolutely needs to be
standardised by the IETF.  They are not part of Ivip.  Its up to AAA
and the MM company how they communicate and how MM probes the
reachability of its network.  IETF standards would probably be
desirable, but even if there were such standards, MM and AAA can do
things differently if it suits them.

If the MM company detects that AAA's network is not reachable via one
or more ETRs then its multihoming service restoration algorithm takes
precedence over whatever AAA last requested in terms of load-sharing
- so the MM system would change the mapping of micronets to use the
ETRs which are working.

In principle, AAA could directly control its mapping itself.  But
this looks tricky to do in order to achieve rapid detection and
mapping changes to achieve multihoming service restoration.  So I
think companies such as MM will be the best approach for most networks.

AAA pays its ISPs for connectivity, and perhaps for use of any ETRs
which the ISPs run.

AAA also pays an annual fee to rent its 6 IPv4 addresses of SPI space
from whichever MAB company it hires them from.   This MAB company is
also responsible for accepting the mapping changes AAA orders, or
which MM orders, and sending these out to all the full-database query
servers (QSDs) in the world.  The MAB company also runs, directly or
indirectly, the DITRs which handle this MAB and so which tunnel
packets to AAA's micronets when they are sent from hosts in networks
without ITRs.  The MAB company would sample this DITR activity and
charge AAA and all its other SPI-space renting customers, according
to the traffic for their micronets which the DITRs handled.

The MAB company either is a RUAS (Root Update Authorisation Server)
company or has a contract with a RUAS company to handle the mapping
for its MABs.  The RUAS companies work together to run the fast-push
mapping distribution system which sends all mapping changes to all
the world's full-database mapping query servers: QSDs

ITRs all over the world get their mapping from local QSDs - directly
or via intermediate, caching QSCs.

Whenever a QSD gets a mapping change for a micronet for which it
recently gave out a mapping, it sends a "map update" message to the
querier.  The querier may be an ITR or a QSD, and if it was a QSD,
the same process occurs.  The result is that ITRs reliably get
mapping changes for micronets they currently have cached mapping for.
 This is explained secured with a nonce from the original map query
and is explained fully in:

  http://tools.ietf.org/html/draft-whittle-ivip-arch

There's no need for AAA's router, or its hosts, to have any ITR function.

The ISPs which it uses (ISP1 for the DSL link, ISP2 for the HFC link)
will typically have their own ITRs.

It is not absolutely essential that these ISPs have ITRs, since if
there is none, then the packets being sent from AAA's network which
are addressed to SPI addresses will go out to the DFZ and then to the
nearest DITR which is advertising the matching MAB.

However, any ISP such as ISP1 or ISP2 which has customers using SPI
space would be motivated to install its own ITR(s) so that whenever a
host of any of its customers sends a packet to an SPI address which
is in a micronet currently mapped to any of the ETRs of these SPI
using customers, these packets will be tunneled to these ETRs from
within the ISP's network, rather than going out to a DITR and coming
back as a tunneled packet to one of these ETRs.

AAA could put an ITR in its network - and it could put ITR functions
in one or more of its sending hosts.  There's no particular reason
for doing so, as long as the ISP provides an ITR which worked fine.

But perhaps in the future an ISP might charge less for customers such
as AAA who didn't emit packets addressed to SPI addresses, and so
burdened the ISP's ITRs.  This seems unlikely, but if this occurred,
it would be an incentive for AAA to install its own ITR(s).

To run ITRs in its network, AAA would need these to be able to
request mapping directly (or indirectly, through QSCs) from one or
ideally two or so QSDs in the ISP's network.

AAA could run its own QSDs, but most small networks would be better
off using their ISP's QSDs, since (with a full deployment, with 10^10
micronets) a QSD will require significant storage (hundreds of
gigabytes) and involve significant inflows of mapping updates.
(Also, at boot time, there is a lot of data to download to the QSD -
though by the time we have 10^10 micronets, a few hundred gigabytes
won't be considered such a large amount of data.)

Whenever an ISP has a customer such as AAA which is using SPI space,
the hosts of the customer will be sending out packets with a source
address which is an SPI address.  The same will be true if AAA runs
its own ITRs, since the encapsulation makes the outer header have the
source address of the sending host.

ISPs will need to accept packets and forward them when they have
these SPI source addresses.   There's no obvious way the ISP could be
fussy about whether or not this particular customer was authorised to
use these SPI addresses - though I suppose it could monitor the
mapping system and see which micronets where currently mapped to the
ETR address this customer is currently using.  But if the ISP was so
restrictive, then this would force AAA to send out packets from an
SPI address N on the same link as whichever is used for the ETR which
currently has N's micronet mapped to it.   This would restrict the
ability of AAA to freely choose its outgoing paths - its outgoing TE.

I previously wrote that an ISP wouldn't be able to know or prevent a
customer such as AAA using its PA address as an ETR, and so using SPI
address.  This is true for the incoming packets (unless the ISP
snooped on the packets going to the customer and saw a bunch of Ivip
IP-in-IP packets).  However, to be useful, AAA really needs its one
or more ISPs to forward packets from these SPI addresses.

Therefore, ISPs should in general forward packets from customers with
any source address matching any MAB.

There's no way the ISP could quickly enough detect that some SPI
address has been mapped to a particular ETR address to alter its
filtering.  It would be perfectly allowable and desirable for some
other end-user network BBB, with some micronet totally unrelated to
AAA's micronets, to map one of its micronets to AAA's ETR - and so
for AAA's hosts to receive these packets and so need to send out
packets with source addresses matching one of BBB's micronets.

The ISP could easily check whether the source address matched any of
the MABs, and therefore was an SPI address - so the ISP could filter
packets emerging from AAA's network to drop any which had source
addresses other than an SPI address, and other than the PA address
the ISP provides with the link.

So it will be trivially easy to spoof SPI addresses.  As far as I
know, it is frequently or typically easy to spoof PI or PA addresses,
so this doesn't represent a significant reduction in security.

I hope these examples can be imagined on larger scales - with whole
corporations and universities using SPI space.

The more end-user networks, of all sizes, we can attract to SPI space
the better for routing scalability.  If the very largest existing PI
using end-user networks keep their PI space, that's OK, because their
prefixes are few in number.  The main aim is to serve the needs of
many networks which currently don't have PI space.

Still, those with PI space could probably get by with less SPI space,
since PI space can only be divided, in IPv4, down to /24s and SPI
space can be divided down to any number of IPv4 addresses.  Hopefully
the largest end-user networks, with large amounts of PI space today,
will find SPI space attractive for some or all their uses - and so be
able to reduce the load on the DFZ by reducing or eliminating their
PI advertisements.

TTR Mobility
------------

This is a brief account of something which is fully documented, with
diagrams, at:

  http://www.firstpr.com.au/ip/ivip/#mobile
  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

This is an IPv4 explanation, but the same thing goes for IPv6 - with
the mobile device getting one or more /64 prefixes.

Let's say AAA has a mobile device, such as cellphone, laptop or
whatever which can get IPv4 access by various means, including 3G
wireless, wired Ethernet, WiFi etc.  (Still, this applies even if it
can only use 3G.)

It doesn't matter that this access may be behind one or more levels
of NAT or if the MN's address uses some Mobile IP techniques.  (It
still works if the MN's address is in SPI address of some network,
including another mobile device.)

AAA decides to make one of its SPI addresses into a micronet and use
it for this mobile device, hereafter known as "MN".

The MN has an ordinary IPv4 stack, but it has an extra bit of
tunneling software with some other features specific to the TTR company.

There could be multiple TTR companies in competition - and AAA
chooses one of them.  I will call it "the TTR company".

The TTR company has a bunch of sites around the world, at various
locations - in major data centres, exchange points or whatever.
There might be two or three in Australia, five or ten in North
America, 15 in Europe etc.   The whole system would work OK with less
than this - I am just describing a fully deployed, mass-market, TTR
company.

In this case, AAA has its own SPI space, but it is quite possible for
a customer to buy a mobile device (PC, cellphone or whatever) and
choose a TTR company which would provide the micronet of SPI space as
well as the services I am about to describe.  The TTR company does
not provide Internet connectivity.  (An ISP could be a TTR company,
but its TTR activities do not provide connectivity.)

Here I assume that only a single IPv4 address is needed for this MN.
 More could be used, and it could use multiple micronets - but one is
used here.

The aim is to give this device this SPI address at all times it has
even a single link to the Internet, no matter where it is.  So this
is a portable, mobile, address.  The MN can still have its
applications send and receive packets from its 3G etc. access network
addresses, but here I assume the applications send and receive
packets using the SPI address.

As soon as the MN has any kind of Internet access, its tunneling
software sets up a two-way, encrypted, tunnel to the last TTR it had
contact with.  (The first time it would start with some central TTR.)

A Translating Tunnel Router (TRR) behaves to the CES system like an
ETR.  The ETR function of the TTR serves multiple MNs at the same
time.  If the MN has a tunnel to a single TTR, then all the packets
addressed to its micronet are tunnelled to the ETR function of the
TTR.  The TTR passes the packets into the bidirectional encrypted
tunnel to the MN.

For now I am assuming a TCP-based tunnel.  In practice, probably a
more sophisticated tunnel would be good.  It would be best to handle
packets of various lengths which may be longer than the TTR to MN
tunnel MTU, by fragmenting them.  This is because the MN might use
various wireless links, with various access networks, and the varying
MTUs of these should not be visible to the sending hosts.

Since the final link is often wireless, the system should be able to
resend packets which do not arrive.  However, some classes of packets
might best be treated like UDP packets, and not resent.

Maybe there is such a tunnel protocol already.

The MN sends its outgoing packets, from its SPI source address, to
the TTR, and the TTR forwards them normally.  Those addressed to
conventional addresses are forwarded to DFZ routers.  Those addressed
 to SPI addresses are tunnelled by the TTR's ITR function.  The TTR's
ITR function needs a nearby QSD, and this would probably be a server
in the same rack.

This will work no matter where the MN is physically, and no matter
which access network it is using.  It can be behind any number of
layers of NAT and it can still tunnel out to the TTR.

To make it work well, the MN should tunnel to a topologically "close"
TTR.  Any TTR in 1000 km or so would be fine - or a few thousand km
if there is none closer.   The TTR company would have some means of
guiding its software in the MN to tunnel to nearby TTRs.

If, for instance in a 3G network, the MN is suddenly given a new
address, it tunnels from that new address to the TTR and
authenticates itself by some means.  There would be a brief loss of
connectivity, but all applications would keep their sessions, since
they are running from the SPI address.  There's no need to change the
mapping when this happens, since the mapping is to the ETR function
of the one TTR.

When the MN has two or more links, it has two or more IP addresses -
in different access networks.  From each such address it would make a
tunnel to the TTR.   This means it can still receive and send packets
if either one of the links fails.  (Exactly how the TTR decides to
send packets is a matter for the TTR company.  It could load-share
the downstream packets over both links, or send duplicate streams for
robustness.  Likewise it is up to the TTR-company software whether it
sends packets up one link, the other, spreads them over both or
whatever.  Either way, they go to the one TTR.

The second address might be topologically distant from the first - so
it might be best to find a different TTR.  Then the MN could make a
tunnel from each address to each TTR.

There could be some tricky software to help the MN find out where its
new address is in the topology and to find the nearest TTR.  But none
of this absolutely needs to be standardised by the IETF, since it is
a matter between the particular TTR company and that company's
software it has provided for the MN.   Ideally this could be
standardised, but there should be room for innovation by various TTR
companies in how they do all this.

The result is that the MN retains its micronet of SPI space, maybe
just a single IPv4 address, no matter where it is - and that by
choosing closer TTRs and then dropping the tunnels to TTRs which used
to be close, but are now distant, the path lengths are generally not
excessively long.

The mapping of the micronet is controlled by the TTR company.  The
TTR company only changes it when it decides that a new TTR is the
best one to send incoming packets to the MN.

An MN could use the same TTR for months or years if it never moved
more than 1000 km or so.   It could still use that TTR if it moved to
the other side of the world, but there would be longer paths, more
latency and more chance of packets being lost.

So TTR Mobility does not mean frequent mapping changes.  Most MNs
wouldn't need their mapping changed from one month or year to the
next.  Only by taking a cross-continent or other long flight would
the MN move enough to warrant choosing a new TTR.

LISP with its slow mapping would work pretty much as well as Ivip for
the TTR mobility architecture.  Ivip would be better, since a new TTR
could be used within a few seconds, and then the old one wouldn't be
needed any more, and the MN could drop its tunnels to the old TTR.

The SPI space the MN gets is global unicast space.  It can be used to
communicate normally with all other hosts - including those using SPI
space, via ordinary ETRs or via TTRs.  The applications in the MN are
perfectly normal and can use any protocols.  The stack itself is an
ordinary IPv4 stack, with tunneling software providing a new IP
address in addition to those which come from the physical connections.