Subject: [rrg] Summary of Ivip for discussion and selection process

From: Robin Whittle <[email-masked]>
Date: Thu, 17 Dec 2009 00:15:29 +1100

Summary of Ivip for the RRG final discussion and selection process
http://www.firstpr.com.au/ip/ivip/Ivip-RRG.html  Robin Whittle 2009-12-17

Key ideas

  Summary

    Ivip (pr. eye-vip, est. 2007-06-15) is a core-edge separation scheme
    applicable to IPv4 and IPv6.  It provides multihoming, portability of
    address space and inbound traffic engineering for end-user networks of
    all types: those with current PI space, those which need their own
    address space but which can't afford PI space and networks or devices
    (mobile devices) which need just a single IPv4 address or /64 of IPv6
    space.

    Ivip's mapping system is based on fast-push of all mapping changes to
    local (within the ISP or end-user network) full database query servers.
    Consequently, ITRs reliably and quickly gain the mapping information they
    need, and all traffic packets are tunneled to the correct ETR without
    loss or significant delay.

    In contrast to other core-edge separation schemes, Ivip's ITRs are given
    a single ETR address for each range of mapped address space (which need
    not be a binary-boundary subnet).  Ivip ITRs do no reachability testing.
    The mapping for a given range of address space is updated for each ITR
    which needs it, in real-time - within a few seconds at most.

    End-user networks are responsible for mapping changes and would typically
    contract this function to specialized companies which monitor the
    reachability of their ETRs and change the mapping to achieve multihoming
    failure recovery and/or TE.  So the mechanisms which choose the mapping
    are directly controlled by the end-user networks, and are completely
    separate from the core-edge separation scheme itself.

    Ivip meets all the constraints imposed by the need for widespread
    voluntary adoption [1].

  Main mechanisms

    ITRs can be dedicated devices, including servers or hardware-based
    routers.  The ITR function can be integrated into the sending host. [2]

    Ivip-mapped end-user address space ("EID" in LISP terminology) can be
    divided into separately mapped ranges, on IP address boundaries (IPv4)
    or /64 boundaries (IPv6).

    The global fast-push mapping distribution network has a structure like a
    cross-linked multicast tree (although other methods could be used).
    All mapping changes are sent to the full database Query Servers (QSDs) in
    each ISP or end-user network which has ITRs.  (Each such network would
    typically have two or more QSDs, each with a different pair of mapping
    feeds from different parts of the fast-push system.)  Each mapping change
    typically incurs a small fee, such as a few cents or tens of cents, which
    will deter frivolous changes and help pay for much of the mapping
    distribution system.

    Compared to conventional unscalable BGP techniques, and to the use of
    core-edge separation architectures with non-real-time mapping systems,
    end-user networks will be able to achieve more flexible and responsive
    inbound TE.  If inbound traffic is split into several streams, each to
    addresses in different mapped ranges, then real-time mapping changes can
    be used to steer the streams between multiple ETRs at multiple ISPs - to
    any ETR in the world.

    Open ITRs in the DFZ (OITRDs, similar to LISP's PTRs) tunnel packets sent
    by hosts in networks which lack ITRs.  So multihoming, portability and
    TE benefits apply to all traffic.  Ivip-mapped space covering the address
    needs of up to hundreds of thousands of end-user networks are contained
    within a single BGP-advertised prefix.  OITRDs around the Net advertise
    such prefixes, gathering packets from non-upgraded networks to the
    closest (in DFZ terms) OITRD.  OITRDs are financed by traffic-based fees
    payable by the end-user networks whose address is mapped.  OITRDs are
    operated directly or indirectly by the individual mapping distribution
    organizations which collectively operate the global fast-push mapping
    distribution system.

    ITRs request mapping either directly from a local QSD or via one or more
    layers of caching query servers (QSCs) which in turn request it from a
    local QSD.  (QSCs are not required, but will reduce the number of queries
    made to each QSD.)  QSDs remember which ITR or QSC requested mapping of a
    range of end-user space.  If, during the caching time of the mapping
    reply, the QSD receives from the fast-push system a changed mapping for
    this range of addresses, the QSD sends a mapping update to the one or
    more ITRs or QSCs which requested this mapping.  The QSCs likewise send
    this update to whichever ITRs or QSCs requested this mapping.  These
    updates are secured by nonce sent in map request and will reach the ITRs
    which need them a few tens of milliseconds after the QSD receives
    the mapping change.

    ETRs may be in ISP or end-user networks.  IP-in-IP encapsulation is used,
    so there is no UDP or any other header.  PMTUD (Path MTU Discovery)
    management with minimal complexity and overhead will handle the problems
    caused by encapsulation, and adapt smoothly to jumboframe paths becoming
    available in the DFZ.  The outer header's source address is that of the
    sending host - which enables existing ISP BR filtering of source
    addresses to be extended to Ivip-encapsulated packets by the simple
    mechanism of the ETR dropping packets whose inner and outer source
    address do not match.

    An end-user network would typically pay rent for their mapped address
    space, which they could split dynamically into any number of individually
    mapped ranges.  They would also pay a fee per mapping update and fees for
    the traffic their network brings through OITRDs.  These services may be
    provided by a single company.  End-user networks would typically
    subcontract control of the mapping of their address space to a specialized
    company, perhaps the same as the one they rent their space from.

    In addition, end-user networks pay one or more ISPs for Internet access.
    They could use an ETR at the ISP, which could be shared by other end-user
    networks.  Alternatively, they could run their own one or more ETRs and
    connect each one to the Net via a small amount of PA space provided by
    their ISP - perhaps a single IPv4 address.

  Extensions

    TTR Mobility

       Ivip would be a good basis for the TTR approach to mobility [3].  This
       is a scalable approach to IPv4 and IPv6 mobility in which the MN keeps
       its own Ivip-mapped IP address(es) no matter how or where it is
       physically connected, including: on conventional PA or PI space, on an
       address which is implemented via conventional Mobile IP techniques or
       on the Ivip-mapped address space of an end-user network - in all cases
       including behind one or more layers of NAT.

       Path-lengths are typically optimal or close to optimal and the MN
       communicates normally with all other non-mobile hosts (no stack or
       app changes), and of course other MNs.

       ETR and ITR functions are combined in a Translating Tunnel Router
       which is located close to, or perhaps within, access networks.  MNs
       establish two-way tunnels to one or more nearby TTRs, but can still
       use TTRs which were formerly close, but are now distant.  A new TTR
       would typically only be desirable if the MN moves more than 1000km.
       Mapping only needs to change when the MN wants to receive incoming
       packets from a new TTR.  No mapping changes are required when the MN
       changes one or more of its physical addresses, since it establishes
       a new tunnel to its one or more current TTRs from each new address.

    Modified Header Forwarding

       Separate schemes for IPv4 and IPv6 enable tunneling from ITR to ETR
       without encapsulation.  This will remove the encapsulation overhead
       and PMTUD problems.  Both approaches involve modifying all routers
       between the ITR and ETR to accept a modified form of the IP header.
       These schemes require new FIB/RIB functionality in DFZ and some other
       routers but do not alter the BGP functions of DFZ routers.

Gains

  Amenable to widespread voluntary adoption due to no need for host changes,
  complete support for packets sent from non-upgraded networks and no
  significant degradation in performance.  Only core-edge separation schemes
  with local full-database query servers can ensure that ITRs tunnel the
  initial packets in a new communication flow without dropping them and with
  delays of no more than a few tens of milliseconds.

  Minimal encapsulation overhead.  In the long term it will be possible to
  migrate Modified Header Forwarding to eliminate encapsulation overhead and
  the need for PMTUD management.  Depending on the time-scale, it may be
  possible to upgrade the routers before introducing Ivip - and so to
  introduce it on IPv4 and/or IPv6 with Modified Header Forwarding.  This
  would make ITRs and ETRs simpler since there would be need for PMTUD
  management.)

  Complete, modular, separation of the control of ITR tunneling behavior from
  the ITRs and the core-edge separation scheme itself: end-user networks
  control mapping in any way they like, in real-time.  Therefore, ITRs
  and ETRs can be relatively simple.  The mapping data itself is much
  simpler than for other core-edge separation schemes: just the start and end
  address of the range to be mapped, and a single ETR address.  There is no
  need for multiple ETR addresses, priorities or weightings - and no need for
  ITR-ETR communication except occasionally for longer packets for the
  purpose of PMTUD management.

  While the TTR Mobility approach can be used with any core-edge separation
  scheme, Ivip's real-time mapping system enables all ITRs to tunnel packets
  to a newly selected TTR in a few seconds.  So the MN could drop the tunnel
  to the old TTR within a few seconds.  An architecture with a non-real-time
  mapping system would need to wait minutes or tens of minutes for cached
  mapping in ITRs to expire before the MN could stop tunneling to the old
  TTR.

  A small fee per mapping change deters frivolous changes and helps pay for
  pushing the mapping data to all QSDs.  End-user networks who make frequent
  mapping changes for inbound TE, should find these fees attractive
  considering how it improves their ability to utilize the bandwidth of
  multiple ISP links.

  Fees for OITRD usage provides a business model for OITRD deployment and
  avoids unfair distribution of the costs of handling packets from non-
  upgraded networks.

  Existing source address filtering arrangements at BRs of ISPs and end-user
  networks are prohibitively expensive to implement directly in ETRs, since
  for larger numbers of addresses, they can only be implemented with
  expensive hardware (power-hungry TCAM).  Ivip's outer header's source
  address is the same as the sending host's address, and simple ETR logic,
  without any need for configuration or knowledge of the BR filtering
  arrangements, enforces the filtering on all decapsulated traffic packets.

Costs

  QSDs receive all mapping changes and store a complete copy of the mapping
  database.  Some fancy protocols and a global network of servers will be
  required to push this information robustly, securely and rapidly to the
  QSDs.  Also, the QSDs need to access centralized servers to initially load
  their database and recover from lost update packets.  However this approach
  eliminates the cost of developing and running an architecture with more
  complex mapping in which ITRs have to choose which ETR to use.

  There is inefficiency in each QSD receiving and storing all mapping, most
  of which will never be needed by any one QSD.  However, this avoids the
  major problem faced by architectures with a global distributed query server
  mapping system: dropping or significantly delaying initial packets while
  the ITR awaits its map reply.

  While mapping data is compact, scaling the system to 10^9 or 10^10
  separately mapped ranges of addresses looks daunting now, but will not be
  such a challenge by the time these numbers arise.  The worst case is 10^10
  IPv6 mappings of about 32 bytes each (start /64, end /64 and ETR address).
  This totals 320GB and fits in a consumer hard drive today.  It will
  probably not be a problem for ordinary server or PC RAM by the time such
  adoption (almost all of it for mobile devices) is achieved.

  It is possible that in some DFZ failure modes, where ETR-A is reachable
  from some parts of the Net and ETR-B is reachable from other parts, that a
  system such as LISP or APT will achieve better multihoming connectivity,
  due to each ITR individually deciding which ETR to use.  During such a
  state, Ivip can't achieve full connectivity, since all ITRs are tunneling
  either to ETR-A or to ETR-B.  However, such conditions are likely to be
  transitory and the LISP/APT ITRs may take some time to make their
  decisions correctly.

Documents

      http://www.firstpr.com.au/ip/ivip/
      http://www.firstpr.com.au/ip/ivip/Ivip-summary.pdf
      http://www.firstpr.com.au/ip/ivip/pmtud-frag/
      http://tools.ietf.org/html/draft-whittle-ivip-db-fast-push-01
      http://tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw-01
      http://www.firstpr.com.au/ip/ivip/ivip6/

 [1]  http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

 [2]  At present, not behind NAT, but if the host creates a two-way tunnel to
      one or more QSCs/QSDs, it can contain an ITR function even when behind
      one or more layers or NAT.  It would be possible to put an ITR function
      in a mobile or poorly connected sending host, but it would be better
      to use an external, well-connected, ITR in these cases.

 [3]  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

---

From: Tony Li <[email-masked]>
Date: Wed, 16 Dec 2009 11:57:13 -0800

> http://www.firstpr.com.au/ip/ivip/Ivip-RRG.html  Robin Whittle 2009-12-17

Robin,

Received, however you've exceeded the 1000 word limit.  Please resubmit.

Thanks,
Tony

---

From: Robin Whittle <[email-masked]>
Date: Fri, 18 Dec 2009 11:48:08 +1100

Hi Tony,

Here's a 995 word version.

  - Robin

Ivip Summary  Robin Whittle 2009-12-18

http://www.firstpr.com.au/ip/ivip/Ivip-RRG-1kw.html

The 2100 word http://www.firstpr.com.au/ip/ivip/Ivip-RRG.html has more
explanation and qualifications.

Key ideas

    Ivip (pr. eye-vip, est. 2007-06-15) is a core-edge separation scheme
    for IPv4 and IPv6.  It provides multihoming, portability of address
    space and inbound traffic engineering for end-user networks of all
    sizes and types, including those of corporations, SOHO and mobile
    devices.

    Ivip's global fast-push mapping distribution network is structured
    like a cross-linked multicast tree.  This pushes all mapping changes
    to full database query servers (QSDs) within ISPs and end-user
    networks which have ITRs.  Each mapping change is sent to all QSDs
    within a few seconds.

    ITRs gain mapping information from these local QSDs within a few tens
    of milliseconds.  QSDs notify ITRs of changed mapping with similarly
    low latency.  ITRs tunnel all traffic packets to the correct ETR
    without significant delay.

    Ivip's mapping consists of a single ETR address for each range of
    mapped address space.  Ivip ITRs do not need to test reachability to
    ETRs because the mapping is changed in real-time to that of the
    desired ETR.

    End-user networks control the mapping, typically by contracting a
    specialized company to monitor the reachability of their ETRs and
    change the mapping to achieve multihoming and/or TE.  So the
    mechanisms which control ITR tunneling are controlled by the end-user
    networks in real-time and are completely separate from the core-edge
    separation scheme itself.

    ITRs can be implemented in dedicated servers or hardware-based
    routers.  The ITR function can also be integrated into sending hosts.
    ETRs are relatively simple and only communicate with ITRs rarely -
    for Path MTU management with longer packets.

    Ivip-mapped ranges of end-user address space need not be subnets.
    They can be of any length, in units of IPv4 addresses or IPv6 /64s.

    Compared to conventional unscalable BGP techniques, and to the use of
    core-edge separation architectures with non-real-time mapping
    systems, end-user networks will be able to achieve more flexible and
    responsive inbound TE.  If inbound traffic is split into several
    streams, each to addresses in different mapped ranges, then real-time
    mapping changes can be used to steer the streams between multiple
    ETRs at multiple ISPs.

    Open ITRs in the DFZ (OITRDs, similar to LISP's PTRs) tunnel packets
    sent by hosts in networks which lack ITRs.  So multihoming,
    portability and TE benefits apply to all traffic.

    ITRs request mapping either directly from a local QSD or via one or
    more layers of caching query servers (QSCs) which in turn request it
    from a local QSD.  QSCs are optional but generally desirable since
    they reduce the query load on QSDs.

    ETRs may be in ISP or end-user networks.  IP-in-IP encapsulation is
    used, so there is no UDP or any other header.  PMTUD (Path MTU
    Discovery) management with minimal complexity and overhead will
    handle the problems caused by encapsulation, and adapt smoothly to
    jumboframe paths becoming available in the DFZ.  The outer header's
    source address is that of the sending host - which enables existing
    ISP BR filtering of source addresses to be extended to encapsulated
    traffic packets by the simple mechanism of the ETR dropping packets
    whose inner and outer source address do not match.

  Extensions

    TTR Mobility

       The TTR approach to mobility [2] is applicable to all core-edge
       separation techniques and provides scalable IPv4 and IPv6 mobility
       in which the MN keeps its own mapped IP address(es) no matter how
       or where it is physically connected, including behind one or more
       layers of NAT.

       Path-lengths are typically optimal or close to optimal and the MN
       communicates normally with all other non-mobile hosts (no stack or
       app changes), and of course other MNs.  Mapping changes are only
       needed when the MN uses a new TTR, which would typically be if the
       MN moved more than 1000km.  Mapping changes are not required when
       the MN changes its physical address(es).

       Separate schemes for IPv4 and IPv6 enable tunneling from ITR to
       ETR without encapsulation.  This will remove the encapsulation
       overhead and PMTUD problems.  Both approaches involve modifying
       all routers between the ITR and ETR to accept a modified form of
       the IP header.  These schemes require new FIB/RIB functionality in
       DFZ and some other routers but do not alter the BGP functions of
       DFZ routers.

Gains

  Amenable to widespread voluntary adoption due to no need for host
  changes, complete support for packets sent from non-upgraded networks
  and no significant degradation in performance.

  Modular separation of the control of ITR tunneling behavior from the
  ITRs and the core-edge separation scheme itself: end-user networks
  control mapping in any way they like, in real-time.

  A small fee per mapping change deters frivolous changes and helps pay
  for pushing the mapping data to all QSDs.  End-user networks who make
  frequent mapping changes for inbound TE, should find these fees
  attractive considering how it improves their ability to utilize the
  bandwidth of multiple ISP links.

  End-user networks will typically pay the cost of OITRD forwarding to
  their networks.  This provides a business model for OITRD deployment
  and avoids unfair distribution of costs.

  Existing source address filtering arrangements at BRs of ISPs and
  end-user networks are prohibitively expensive to implement directly in
  ETRs, but with the outer header's source address being the same as the
  sending host's address, Ivip ETRs inexpensively enforce BR filtering on
  decapsulated packets.

Costs

  QSDs receive all mapping changes and store a complete copy of the
  mapping database.  However, a worst case scenario is 10 billion IPv6
  mappings, each of 32 bytes, which fits on a consumer hard drive today
  and should fit in server DRAM by the time such adoption is reached.

  The maximum number of non-mobile networks requiring multihoming etc. is
  likely to be ~10M, so most of the 10B mappings would be for mobile
  devices.  However, TTR mobility does not involve frequent mapping
  changes since most MNs only rarely move more than 1000km.

Documents

 [2]  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

---

From: Tony Li <[email-masked]>
Date: Sun, 20 Dec 2009 11:01:46 -0800

> Here's a 995 word version.

Received and incorporated.  Thanks.

Tony