Subject: [rrg] Constraints on a solution - "incrementally deployable" again

From: Robin Whittle <[email-masked]>
Date: Thu, 16 Apr 2009 12:40:53 +1000

Short version:  I hope many people will agree with something like:

                A successful solution to the routing and addressing
                scaling problem is tightly constrained by the need
                for it to be voluntarily adopted by the great
                majority of end-user networks of all sizes who want
                or need multihoming, inbound traffic engineering
                and/or portability of their address space between
                ISPs.

                Broadly speaking, this means that at the time of
                introduction:

                 1 - The solution must provide immediate and
                     compelling benefits to the end-user networks
                     who adopt it, irrespective of how many other
                     networks have so far adopted it.

                 2 - Therefore, it must provide multihoming, TE and
                     portability for the adopting networks in a
                     manner which fully supports communications with
                     all hosts - including all hosts in non-upgraded
                     networks.

                 3 - Therefore, the solution must be compatible with
                     all hosts (stacks and applications) and routers
                     in non-upgraded networks.

                 4 - The solution must be compatible with all
                     existing applications in the adopting networks.

                 5 - Although in principle, some or many host stacks
                     could be upgraded in the adopting networks, the
                     difficulties this presents means that a scalable
                     routing solution will only be widely enough
                     adopted on a voluntary basis if no such upgrades
                     are required in order for the new system to
                     provide compelling benefits.

                     Firstly, there are difficulties in motivating
                     operating system / stack programmers to add and
                     test new facilities.  Secondly there may be
                     costs and disruption for the adopting network in
                     upgrading the stacks.  Thirdly, in many or most
                     end-user networks, some or many hosts will not
                     be amenable to stack upgrades due to them using
                     operating systems which are no longer being
                     maintained or developed.  This is especially
                     the case for devices such as printers, routers
                     etc. which do not use one of the major operating
                     systems.

                 6 - The solution must be compatible with all
                     existing DFZ routers.  (Unless it can be shown
                     that all DFZ routers can be upgraded - such as
                     via a software update - by the time of
                     introduction.)  [See note at end of message.]

                 7 - The solution must be attractive for the large
                     number of smaller networks who currently cannot
                     get PI space to do multihoming etc.  Our goal is
                     not just to reduce the number of DFZ routes but
                     to enable a much larger number of end-user
                     networks to gain multihoming, TE and portability
                     than is currently possible.

                 8 - Ideally the solution will also be attractive for
                     most or all end-user networks who currently have
                     their own PI prefixes.  This is firstly to
                     improve scaling by attracting those larger
                     end-user networks away from their current
                     conventional PI address space usage and secondly
                     to ensure that smaller networks which aspire to
                     be bigger in the future will perceive the new
                     kind of scalable end-user network address space
                     as suitable for their long-term needs.

                 9 - Ideally the new system should not compromise
                     security or performance compared to that of
                     current PI or PA techniques.  Any degradation
                     in security, packet path lengths, end-to-end
                     packet transit times, packet loss rates and
                     general robustness etc. needs to be more than
                     compensated for by reduced costs and/or benefits
                     such as a more flexible and responsive
                     multihoming restoration capability than is
                     possible with current BGP techniques.

                10 - The solution will not be widely enough adopted
                     if it requires upgrades to the internal routers
                     of adopting networks - except perhaps in very
                     small networks such as SOHO where one or a few
                     routers need upgrading anyway to perform ITR and
                     ETR etc. functions.

                In the longer term - in the years and decades to
                come - upgrades to DFZ routers, to ISP and end-user
                network internal routers and to host stacks can be
                made without significant cost or inconvenience and
                may allow enhanced performance for the new system.

I counted 4 people, including myself, supporting my statement about
the scalable routing problem in the "Re: Point of order" thread:

  http://www.ietf.org/mail-archive/web/rrg/current/msg04786.html

    It is a problem whose solutions are tightly bound by the need for
    compatibility with all existing hosts and most existing routers -
    and by the need to ensure immediate benefits for early adopters,
    since the solution will only work if it is voluntarily adopted by
    most end-user networks which want multihoming, portability etc.

Xiaoliang Zhao (4789), Dino Farinacci (4794) and John Zwiebel (4801).

Joel, you are the only one so far to disagree:

  http://www.ietf.org/mail-archive/web/rrg/current/msg04797.html

> If "it" is a solution to the problems of routing scalability,
> reliability, robustness, etc that the rrg was chartered to work on,
> then I do not believe we ever agreed that compatibility with all
> existing hosts was a requirement.

I wasn't suggesting we had all agreed to this, or that "rough
consensus" on this had been achieved.

The sentence I wrote was a terse version of what I believe to be
true.  I believe it would be a good idea to reach rough consensus on
something like this, such as the new text above.

> Deployability in such a way that existing hosts and existing
> routers can interwork with any solution, deployability such that
> there is value in incremental deployment, and migratability so that
> incremental deployment is possible are all things I would buy into.

I agree with this, according to my understanding of "incremental
deployment" - but after much debate from 2007-11-07 to 2008-04-10 I
discovered that my understanding of this term could not be defined in
a few words, and was considerably more complex than the meaning
several other people ascribed to this term.

You contributed to the debate on 2008-03-18:

  http://www.ietf.org/mail-archive/web/rrg/current/msg01775.html

> One small aspect that I think I am seeing here is the same words
> ("incrementally deployable") being used to describe two different
> concepts. I may be confused, but it looks to me like:
>
> Deployable incrementally without disruption: This means that a
> portion of the net (host, site ISP, whatever) can deploy the system
> without losing capability or connectivity. A flag day is not
> necessary in order to get the system operational.
>
> Deployment incremental incentives: This sounds like what Robin is
> asking for, and is related to a bunch of work I have seen from
> folks like Dr. Odlyzko on the economic incentives that are needed
> to drive deployment. It includes analysis of issues like when do
> folks need positive return on new technologies to cause them to
> deploy.
>
> They are both valid concerns. But I think they are two different
> dimensions.

I agree entirely.  I realised that my use of the term "incrementally
deployable" involved all the conditions for people being motivated
to adopt it - in particular by the scalable routing solution
providing real benefits for the adopters even when very few other
networks have so far adopted it.  I think this means the system must
provide multihoming, TE and portability of address space in a way
which works with all incoming packets, including those from
non-upgraded networks.

Brian Dickson responded to your message:

  http://www.ietf.org/mail-archive/web/rrg/current/msg01881.html

raising 15 separate points in 4 categories, including:

    We should be sure the benefits are truly compelling, ideally
    universally so, and the costs so marginal as to be nearly
    negligible, before settling on a recommendation. That may
    mean more work for us, but it is a small price to pay.

which I entirely support.

You also wrote (2008-03-20):

  http://www.ietf.org/mail-archive/web/rrg/current/msg01848.html

> I realize that there has to be a limit how far down the economic /
> business analysis we go. Among other things, we need to start by
> coming up with ideas, without regard to such limits. But when it
> comes to deployment analysis, ignoring the question of who pays,
> and why, is just not going to work. We don't need (don't want, and
> can't) to mandate a business model. I would hope that a sensible
> recommendation allows for multiple business models. But if we can
> not even imagine one that works, then it fails.

which I also entirely support.

On 2008-04-02 I came up with a detailed description of the deployment
requirements of a successful scalable routing solution:

  http://www.ietf.org/mail-archive/web/rrg/current/msg02030.html

======

     "No significant barriers to widespread deployment due to
      initially low uptake rate, or disruption of existing
      practices"

or:  "The zero (or relatively low) critical mass criteria for
      ubiquitous deployment on an incremental basis without
      outside inducements"

  (10) The technology must be able to be introduced without
       disrupting existing practices.  After some low
       specified level of initial deployment, ideally zero,
       everyone (or at least most people) who deploys the
       technology receives benefits (maybe net benefits in
       some short enough time-frame after accounting for
       one-off purchase and installation costs) which are
       not only positive, but substantial and likely to
       attract many other end-users to the technology.
       For this to be true, the benefits received must not
       depend much or at all on what proportion of other
       users have adopted the technology.  Therefore, if the
       technology would be fundamentally attractive to many,
       most or all end-users once deployed by most or all
       end-users, it would also be attractive enough to the
       very first (or after some low, specified, level of
       adoption) to actually become widely deployed via a purely
       incremental means, without inducements or a major up-front
       deployment before any end-users gain significant benefits.

======

That's a bit of a mouthful, but it is quite a task to fully describe
the constraints of upgrading the routing and addressing system of the
Internet when we are relying on purely voluntary adoption by end-user
networks and their ISPs.

The text at the start of this message is an attempt to rewrite the
sentence Xiaoliang Zhao, Dino Farinacci and John Zwiebel supported,
in the hope that you and many others will support it too.

You wrote:

> And I do not agree that getting clear terminology is irrelevant to
> getting to such a goal.  I have frequently found it very confusing,
> and a hinderance to progress, to realize that the person I was
> talking with meant something different by the terms he was using
> than I meant by them.  Yes, it happens.  But that does not make it
> helpful.  Getting a set of terms we can agree upon, and using them,
> can be very helpful.

Yes, but I do not want to try to define exact semantics for
"incrementally deployable".  I am wary of using the term at all,
without explicit definition, because it apparently means different
things to different folks, as indicated by the debate over a year ago:

http://www.google.com/search?q=RRG+"incrementally+deployable"+site%3Awww.ietf.org+-"rrg+Discussion+Archive"

  - Robin

Note regarding upgrading DFZ routers:

   If all DFZ routers (or every DFZ router between any ITR and
   any ETR) can be replaced or upgraded with a firmware update
   before the solution is deployed, then we could gain
   considerable benefits.

   There may be other benefits of this, but I am thinking of
   Modified Header Forwarding as an alternative to encapsulation
   - which would eliminate encapsulation overheads and greatly
   reduce ITR and ETR complexity, due to avoiding the Path MTU
   Discovery problems which are inherent in encapsulation:

     http://tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw
     http://www.firstpr.com.au/ip/ivip/ivip6/

   How soon must a solution be deployed?

   The IPv4 DFZ routing table http://bgp.potaroo.net currently has a
   doubling time of about 5.3 years.  I think it will be 5 years or
   more before we really try to deploy a scalable routing system for
   IPv4.

   IPv6 DFZ prefixes (click number for IPv6 prefix count at
   http://bgp.potaroo.net/v6/v6rpt.html) currently number about
   855 which is 1/256 of the 219,603 for IPv4.  The doubling
   time is between 2.7 and 4 years.  Even with a doubling time of
   2 years, it would take 16 years to reach the current IPv4
   numbers.

   So I think it will be perfectly feasible to have IPv6 DFZ
   routers upgraded for Modified Header Forwarding in time
   for a scalable routing solution.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 15 Apr 2009 23:18:56 -0400

This long list of "goals" has the interesting property that each of them 
sounds reasonable, but each is phrased in such a way as to leave me 
wondering if it is intended to imply cases or interpretations with which 
I disagree.

To use your original text as an example, "compatibility with all hosts" 
seems to imply that the solution can not start by modifying some of the 
hosts.  I know that you have argued that a solution based on host 
modification is untenable, and this is a topic on which we have disagreed.

I absolutely agree that deployment has to have value for the deployer, 
in a reasonable and practical deployment scenario.  And the deployer 
must not lose the ability to talk to the rest of the world in exchagne 
for that value.  Most of the implications I see do not quite match up 
with the way you chose to phrase things below.

An example in the first paragraph below is that the question of what the 
deployment path is, and when the bulk of users see value, and what value 
they see, varies depending upon the solution.  Nothing will be 
"voluntarily adopted by the great majority of ..." early in its 
introduction.  The path to widespread adoption is tricky for any technology.

Yours,
Joel

---

From: Tom Vest <[email-masked]>
Date: Wed, 15 Apr 2009 23:55:38 -0400

Hi Robin,

FWIW I like what's here. However, if we subsequently find ourselves in  
a world where IPv6 remains undeployed, and (possession of) IPv4  
continues to represent a critical bottleneck to nontrivial  
participation in the routing system, then most/all of the maxims below  
would seem to be largely or exclusively relevant to "incumbent" IPv4  
holders. If I'm not off-base in this observation, it would have two  
troubling implications. First, it suggests that this particular vision  
of incremental deployability foregoes, if not excludes, one of the key  
drivers of technology change in other sectors -- i.e., the possibility  
of early adoption led by aspiring new entrants, in this case aspiring  
routing services providers. Second, while the recommendations should,  
if followed, increase the odds of the associated technology being  
widely deployed, it would still leave the fate of the technology in  
question to be determined absolutely by "incumbent" routing system  
participants. In this specific sense, a future technology that  
satisfies all of the requirements for "incremental deployability"  
might still end up getting effectively "vetoed," in the way that (some  
say) IPv6 has been vetoed.

Before I even attempt to suggest what kind of extensions or  
modifications might serve to remedy this perceived "incumbent bias"  
I'd be very interested in hearing reactions from others...

Thanks,

Tom Vest

On Apr 15, 2009, at 10:40 PM, Robin Whittle wrote:

> http://www.google.com/search?q=RRG+"incrementally+deployable"+site 
> %3Awww.ietf.org+-"rrg+Discussion+Archive"
>
>  - Robin
>
>
> _______________________________________________
> rrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/rrg

---

From: Robin Whittle <[email-masked]>
Date: Thu, 16 Apr 2009 14:48:44 +1000

Short version:  Responding to Joel, with a change to the first
                paragraph.

Hi Joel,

> wondering if it is intended to imply cases or interpretations with which

I am glad you think each point sounds reasonable.  I am not trying to
slip anything by you.  Please let me know how you might disagree with
possible interpretations of the various points.

> modification is untenable, and this is a topic on which we have disagreed.

The new text is not so absolute.  All 10 points are "Broadly speaking
. . ." which I intend to meaning something like "These are good
general principles - and that any deviation from them would require a
convincing argument".

Please give an example of how a scalable routing solution could be
widely adopted (or adopted at all by end-user networks in general, in
the early stages) if it required some form of host changes in order
for it to deliver compelling benefits for the adopting networks.  For
instance:

 1 - Changes to host stacks or applications in other networks.
     (A non-starter, I think we all agree.)

 2 - Changes to all applications in the adopting network.
     (Likewise, a non-starter, I am sure.)

 3 - Changes to the some or all applications in the adopting network.

 4 - Changes to stacks in some or all hosts in the adopting network.

Please give examples of why such changes might be an essential
element of any scalable routing solution which would be generally
superior to alternative solutions which require no such changes.
Please also give some examples of how particular types of end-user
network might be motivated to make such changes.

I am not opposed to host changes on an optional basis.  For instance,
Ivip has optional ITR functions in sending hosts - not behind NAT,
but on ordinary ("RLOC") or Ivip-mapped micronet ("EID") addresses.
My concern is about a scalable routing solution requiring host
changes in some or all hosts in order for it to deliver the
compelling benefits needed for widespread voluntary adoption.

> for that value.  

I think we all agree on this.

> Most of the implications I see do not quite match up with the way
> you chose to phrase things below.

> introduction.  The path to widespread adoption is tricky for any

OK - here is a version which hopefully overcomes this problem - the
addition of "over a period of years".

                 A successful solution to the routing and addressing
                 scaling problem is tightly constrained by the need
                 for it to be voluntarily adopted, over a period of
                 years, by the great majority of end-user networks of
                 all sizes who want or need multihoming, inbound
                 traffic engineering and/or portability of their
                 address space between ISPs.

Please point out any other problems with implications you foresee and
suggest any other things which could be improved.

BTW, I used an invalid address Brian Dickson. I think it should be:

  Brian Dickson <[email-masked]>

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Thu, 16 Apr 2009 15:22:49 +1000

Short version:    Responding to Tom Vest's concern about how the
                  constraints on widespread adoption of a scalable
                  routing solution seem to preclude widespread
                  adoption of IPv6.

Hi Tom,

Regarding my suggested text:

  http://www.ietf.org/mail-archive/web/rrg/current/msg04815.html

you wrote:

> FWIW I like what's here. However, if we subsequently find ourselves in a
> world where IPv6 remains undeployed, and (possession of) IPv4 continues
> to represent a critical bottleneck to nontrivial participation in the
> routing system, then most/all of the maxims below would seem to be
> largely or exclusively relevant to "incumbent" IPv4 holders. 

I tried to describe the constraints as I saw them.  The pressure to
retain existing applications and stacks is extremely strong - surely
too strong in the foreseeable future to allow widespread voluntary
adoption of a solution which required extensive stack or application
changes.

I didn't refer to the possibility of adopting a completely new kind
of new address space (IPv6 instead of IPv4) and the required new ISP
connectivity to support it, as would be the case if the solution
involved IPv6 or some other alternative to IPv4.

That would be precluded by the points:

   2 & 3 - Hosts with the IPv6 address only get multihoming etc. with
           communication with other IPv6 hosts.  Because the solution
           presumably only provides benefits for IPv6 communications,
           it fails to meet this constraint that it "fully supports
           communications with all hosts - including all hosts in
           non-upgraded networks."  since, during adoption, most
           hosts in non-upgraded networks are IPv4 hosts.

   4 & 5   Most end-user networks which want multihoming do not, at
           present, have all their hosts and applications fully ready
           for IPv6.

However, perhaps something needs to be added to the text to address
your concern about how the scope of text seems to be within the
current IPv4 situation, rather than looking beyond it.

> If I'm not
> off-base in this observation, it would have two troubling implications.
> First, it suggests that this particular vision of incremental
> deployability foregoes, if not excludes, one of the key drivers of
> technology change in other sectors -- i.e., the possibility of early
> adoption led by aspiring new entrants, in this case aspiring routing
> services providers. 

I understand you are referring to IPv6 being promoted by existing or
new ISPs who have a vision for IPv6 and promote its adoption.

IPv6 adoption remains glacial:

  http://bgp.potaroo.net/v6/v6rpt.html

        BGP advertised   Autonomous
        prefixes         Systems

  IPv4  219603           25296
  IPv6     855             741

I can't imagine the majority of current Internet users adopting it
any time soon.

Perhaps in the future it will be adopted for fixed networks in
countries where IPv4 space is in too-short supply.  However, I think
that any such service would need IPv4 connectivity to be useful - so
IPv6 addresses for the hosts is not the whole story - the hosts still
need to tunnel to some IPv4 proxy server or whatever, so they can
function on the IPv4 Internet.

The most likely scenario I can imagine for large-scale adoption of
IPv6 is in mobile networks - 3G cellphone networks and whatever they
develop into.  AFAIK, this has not occurred to any substantial degree
yet.

I agree that the constraints as I wrote them seem to preclude
widespread adoption of IPv6, except perhaps in some future mobile
networks where direct connectivity to IPv4 hosts is for some reason
not required.  I think the text reflects the reality, however
unfortunate the reality is for the migration to IPv6.

> Second, while the recommendations should, if
> followed, increase the odds of the associated technology being widely
> deployed, it would still leave the fate of the technology in question to
> be determined absolutely by "incumbent" routing system participants. 

Since we (the RRG, the IETF etc.) have no coercive powers, I agree
that the fate of a particular network is absolutely determined by
those who are running and using the network.

I see our task as requiring the development of a new form of address
space, with all the supporting protocols and software for routers and
other network elements, which will solve the scaling problem if
widely deployed.  Then we need to make that system so attractive that
it will be widely adopted by most end-user networks which want
multihoming etc. - even though they may neither know or care about
the scalability problem, bloated DFZ route numbers etc.  We need to
show it works, that people can make money using it and generally
promote it.  However, only if it works really well will it be widely
enough adopted to solve the routing scalability problem.

> In
> this specific sense, a future technology that satisfies all of the
> requirements for "incremental deployability" might still end up getting
> effectively "vetoed," in the way that (some say) IPv6 has been vetoed.

I am trying not to ascribe any particular meaning to "incremental
deployability".

I don't agree that IPv6 has been "vetoed".  It has been available for
10 years or more an almost no-one really wants it, for the simple
reasons that it doesn't provide connectivity to the great majority of
hosts and that for most people, it provides no benefit over IPv4.

> modifications might serve to remedy this perceived "incumbent bias" I'd
> be very interested in hearing reactions from others...

I think the term "incumbent bias" implies an unreasonable resistance
to change.

I don't think it is unreasonable that the great majority of Internet
users want and need a continuing service which enables them to
communicate with every other Internet host used by other similar people.

The IPv4 Internet is by far the biggest and most significant IT
system which has ever existed.  In a free country, there would be no
demand for a service which failed to connect to even 1 or 2% of
hosts.  An IPv6-only service fails to connect to nearly 100% of hosts.

This is no fault of IPv6 - its just that no-one has figured out how
to remain IPv4 compatible while upgrading to another network.  It
could be done, in principle - but only by radically rewriting all
applications to use a new stack.  Even then, it would be a complex
and error-prone business trying to retain functionality running on
one or the other network exclusively, or using them both at the same
time.

Here is a potential addition to the text, but I think this is taking
the scope of the piece beyond the constraints to adoption, and into
some additional goals we might have regarding the long-term
development of the Net:

                The above text attempts to summarise the real-world
                constraints on a scalable routing solution.  It
                appears that these constraints create a very high
                barrier to migration of significant numbers of
                existing end-user networks from IPv4 to an IPv6
                service, either with or without retaining their
                IPv4 address space.

                It follows that even if IPv6 had a scalable routing
                solution, that the barriers to mass adoption of it
                would remain so high as to preclude a mass migration
                from IPv4.

                Our goal is to solve the routing scaling problem
                in both the IPv4 and IPv6 Internets - and for the
                foreseeable future, the problem only manifests in
                the IPv4 Internet.

                Perhaps an additional goal of a scalable routing
                solution would be - by some methods yet to be
                determined - to facilitate adoption of an alternative
                to IPv4 such as IPv6 which is a more promising
                basis for accommodating very large numbers of
                individual hosts, many of which are expected to be
                mobile devices.

Maybe something like this could be a footnote.

   - Robin

---

From: Tom Vest <[email-masked]>
Date: Thu, 16 Apr 2009 12:37:53 -0400

Hi Robin,

Thanks very much for the detailed response. Further thoughts inline.

>           present, have all their hosts and applications fully ready

If I'm interpreting correctly, it seems that you agree with my  
original surmise, i.e., that
the idea of "incremental deployability" as originally specified is  
completely consistent with an absolute prerequisite of (what is likely  
by the earliest feasible moment of applicability to be) "ownership" of  
IPv4 addresses.

This may be less problematic, I suppose, if the intent is to scope the  
proposed recommendations as "institutional-level incremental  
deployability for current ( aka 'incumbent') routing system  
participants" -- and that scope is made explicit in the text.

However, I originally interpreted the text more broadly, c.f., as  
guidelines intended to facilitate "deployability" more generally,  
toward the presumptive, pragmatic goal of real-world deployment. It  
seems to me that the only domain of conceivable relevance to  
"deployability" as defined by RRG is a built environment that is, as a  
practical matter, composed wholly of deployed routing protocols that  
derive (and/or are deprived of) much if not all of their value by  
virtue of the presence/absence of extra-institutional "network  
effects." An additional, perhaps contingent but nevertheless  
inescapable feature of the real-world deployability environment is the  
absolute need for one specific kind of quantity constrained protocol  
number resource (unique IPv4), the availability of which will shortly  
become (at the very least) highly uncertain for anyone who does not  
already have it.

To my mind at least, this particular combination of environmental  
factors -- i.e., the centrality of specific, established network  
effects plus the de facto inaccessibility of those effects to any  
aspiring routing system participant that does not possess a specific,  
scarce, and potentially unobtainable legacy protocol-defined input --  
imposes some hard constraints on any set of RRG "incremental  
deployability" recommendations. The network effects factor effectively  
makes the utility of any purely institution-level specification of  
"incremental deployability" highly dubious -- or alternately wholly  
contingent on the accommodation of those network effects at the  
individual institution (i.e., routing system participant) level.  
However, recognition of the real-world correlates of those network  
effects entails coming to terms with the inescapable quantitative/ 
demographic constraints that are inherent in IPv4.

My own short version: explicitly acknowledge possession IPv4 as an  
absolute prerequisite, and institutional-level deployability as the  
intended scope of the proposed incrementalism recommendations.  
Alternately, modify as necessary to explicitly accommodate incremental  
deployment paths that are not contingent on possession of IPv4.

> current IPv4 situation, rather than looking beyond it.

That is indeed my principled concern, based on the last couple of  
years of work and research.
More (just a little) on that below...

> I can't imagine the majority of current Internet users adopting it

Actually we are in full agreement on this point. My intuitions are

(1) that IPv6 will not be adopted by enough parties to become viable  
as a stand-alone extension to IPv4 for any entity that does not *also*  
possess IPv4, or exclusive beneficial use rights thereto. By  
implication, any set of prescriptions that defines "incremental  
deployability" without some consideration of the absolute finitude,  
and likely increasing non-availability of unique IPv4 address  
resources, will embed and thereby (potentially) perpetuate the  
requirement for IPv4.

(2) that the present effort to scope "incremental deployability" is at  
least partially motivated by the desire to reduce the odds that future  
RRG efforts will encounter deployability impediments similar in form  
and/or impact to those that have been revealed by the IPv6 experience.

The speculations below about how IPv6 might or might be deployed in  
the future are interesting, and might merit further discussion off- 
list, but are not relevant to the concerns I've expressed here.

> those who are running and using the network.

While I think I understand your point here, it seems to me that this  
particular phrasing is orthogonal to any relationship between RRG and  
incremental deployability. Granted, the IETF has no coercive powers.  
Of course, the fate of anything/everything outside of the scope of  
candidate future routing protocols is beyond RRG's ambit. These  
undeniable facts would clearly support an assertion like:

"...the fate of a particular *network protocol* is absolutely  
determined by those who are using, or who would be able and willing to  
use that particular protocol."

  However, extrapolating beyond that to the fate of "a particular  
network" would only make sense if that "particular network" is *not*  
the environment that imposes deployability criteria (i.e., not the  
Internet as it exists concurrently), or else if one as prepared to  
concede the above points (i.e., possession of IPv4 is assumed, and  
should be understood as an absolute prerequisite for anyone who hopes  
to use this "incremental deployability" formula to actually deploy  
something).

> enough adopted to solve the routing scalability problem.

I agree, but I think these claims only reinforce my point. Given our  
structural dependence on decentralized, privately incentivized action  
as the only vehicle for protocol deployment, your "so attractive"  
benchmark can only be understood in relative/comparative terms. Thus,  
regardless of the specific virtues of the hypothetical future network  
protocol candidate, the actual attractiveness hurdle for existing  
routing system participants will be defined by the private, individual  
operator-level benefits that the candidate protocol provides *over and  
above* the alternatives, including other candidate solutions that  
offer higher short-term private benefits (profits) and/or lower short- 
term direct costs -- as well as the ever-present option of just  
sticking with the status quo.

By contrast, the attractiveness hurdle for aspiring ("non-incumbent")  
routing system participants will be all of the above *plus* the cost  
of acquiring IPv4. In the near future, that could well imply that  
deployment of any kind, incremental or otherwise, will only be  
possible for entities that have achieved a favorable outcome in  
negotiations to purchase scarce, non-substitutable IPv4 space from a  
competitor -- i.e., from a routing services provider that knows that  
the sale will, by definition, enable the buyer to enter the routing  
services market and thus become a potential direct competitor. The  
history of competition in industries in which "bottleneck" (essential  
+ non-substitutable) inputs play a central role suggests that that  
hurdle could be very, very high.

I don't think that RRG is responsible for the fact that IPv4 will soon  
impose these and other bottleneck effects on incremental  
deployability, or on the Internet more generally. However, I have  
reason to believe that it may be extremely difficult (i.e.,  
approaching impossible) to define *any* routing protocol that might  
satisfy that attractiveness test for existing routing system  
participants -- and perhaps absolutely impossible to define an  
acceptable solution that does not *also* reproduce the same general  
demographic constraints that IPv4 will impose.

This is why I feel strongly that any RRG statement on "incremental  
deployability" should not explicitly or implicitly assume possession  
of IPv4 as a prerequisite. My own idiosyncratic experiences lead me to  
believe that the deployability of any/all future RRG outputs may  
absolutely dependent on the ability of non-incumbents to lead and  
drive the adoption dynamic (more on this below). I don't expect the  
official RRG "incremental deployability" guidelines to adopt this  
stance, but neither could I support guidelines which are not equally  
applicable to all potential adopters, regardless of their access to  
IPv4.

>> In this specific sense, a future technology that satisfies all of the

In the real-world Internet where the provider-level value of a routing  
protocol is dictated by inter-provider network effects, an "abstain"  
vote by enough participants is equivalent to a system -wide veto.  
However, I didn't intend to use that term to imply anything more than  
what you stated in your response...

>> I'd

The perception of "incumbent bias" does not arise if the enumerated  
"incremental deployability" recommendations are not in fact contingent  
on possession of IPv4. By implication, an official position of  
"remaining agnostic" on this question would be appropriate IFF the  
final "incremental deployability" recommendations would be *equally*  
relevant, applicable to, and adoptable by both IPv4 haves and have-nots.

> one or the other network exclusively, or using them both at the same

Again, we're in full agreement on these particulars, but as above I  
don't think that our agreement on these points is germane to the  
question at hand.

> Maybe something like this could be a footnote.

I would go further than you and suggest that the proposed amendment  
has nothing at all to do with incremental deployment.

I would much prefer to see the text explicitly acknowledge the fact of  
an IPv4 prerequisite, and all that it implies for incremental  
deployability, and deployability more generally. That said, what I  
would *most* like to see is some reformulation that illuminates  
possible incremental deployability paths that do not proceed from an  
assumption of possession of IPv4.

****

So... for those who've gotten this far, I actually have reasons for  
believing that this modification is critically important. My personal  
experience is a one factor -- I helped to build out and "peer up" what  
was briefly a "Tier One" ISP, with most of my hands-on experience  
accumulated in the course of building out in parts of the world where  
artificially imposed IP addressing constraints were a common tool used  
by locally dominant operators. But much more apropos is the tentative  
findings of an ongoing 5+ year research project (first began as my  
prospective doctoral dissertation), which suggests that the Internet  
is at root a new kind of "liquidity system," one that is limited in  
scope to packetize-able things and which builds on a largely static  
"attachment" mechanism rather than the more dynamic "payment"  
mechanism of that older and more familiar liquidity system (i.e., the  
global system of monetary instruments, financial flows, credit  
creation, etc.).

On this interpretation, the deployability of any future routing/ 
addressing protocol is likely to raise the same issues and impose the  
same challenges that many economies have experienced in the past when  
trying to integrate a new currency, or migrate from one currency to  
another. The existence of sovereign authority as a driver of currency  
migration makes some historical cases less relevant than others, but  
history reveals that even sovereign power cannot assure a successful  
liquidity system adaptation under some conditions. This shouldn't be  
altogether surprising, since the "circulating medium of  
exchange" (ala"money") was not originally invented so much as  
recognized post facto -- following its "emergence", usually in the  
form of some pretty-but-not-terribly-useful naturally occurring  
factor, as a widely acceptable, hence "fungible," temporary proxy for  
the thing(s) that aspiring buyers and sellers are actually trying to  
obtain. But to the main point, the most directly relevant historical  
cases are those in which sovereign power was not relevant, because the  
monetary liquidity system was not governed by a national treasury or  
central bank, but rather was largely self-coordinating through  
decentralized, limited, voluntary, private incentive-driven  
coordination mechanisms -- typically called "banker's clubs."  
Structurally and functionally, the monetary liquidity arrangements  
that characterized these so-called "free banking" periods bear an  
extremely strong resemblance to routing and addressing-related  
features of the TCP/IP-based Internet. And that's cause for serious  
concern.

Anyone's who's still interested can find more details here:
http://tinyurl.com/IPliquidity

Tom

---

From: Robin Whittle <[email-masked]>
Date: Fri, 17 Apr 2009 13:29:02 +1000

Short version:   Slightly amended text, followed by a brief
                 reply to Tom Vest's message and some ideas
                 on how large mobile networks could be the
                 first large-scale IPv6 deployments.

                 Changes indicated: *.

Constraints imposed by the need for voluntary adoption        *

    A successful solution to the routing and addressing
    scaling problem is tightly constrained by the need
    for it to be voluntarily adopted, over a period of        *
    years, by the great majority of end-user networks         *
    of all sizes who want or need multihoming, inbound
    traffic engineering and/or portability of their
    address space between ISPs.

         introduction.)

    10 - The solution will probably not be widely enough     *
         adopted if it requires upgrades to the internal
         routers of adopting networks - except perhaps in
         small networks such as SOHO where one or a few      *
         inexpensive routers need upgrading anyway to        *
         perform ITR and ETR etc. functions.

[Optional footnote if most people like it.]

    As a footnote, it should be acknowledged that the         *
    constraints as described above seem to involve great      *
    difficulty in providing any Internet service at all       *
    - based on scalable routing or not - beyond IPv4.         *

    It follows that even if IPv6 had a scalable routing
    solution, that the barriers to mass IPv6 adoption
    would remain high.  A potential exception may be         *
    mobile devices, whose functionality may not be as        *
    dependent on full IPv4 connectivity as is the case       *
    for most hosts today.    [See note below.]               *

    A good IPv4 scalable routing solution may enable         *
    finer divisions in the address space than is             *
    practical with current BGP techniques and so may         *
    contribute to a significantly higher number of           *
    end-user networks being able to use the available        *
    space.  Nonetheless, the finite limits of IPv4           *
    address space remain a serious concern, especially       *
    for newly established ISPs and end-user networks.        *

    On this basis, it would be desirable if a                *
    scalable routing solution, in some way yet to be         *
    determined, facilitated adoption of an alternative       *
    to IPv4 - such as IPv6 - which is a more promising       *
    basis for accommodating billions of hosts, many of       *
    which are expected to be mobile devices.                 *

Note on IPv6 mobile devices.   I am envisaging a situation, such as
in China or other countries, where cellphones and mobile PCs etc.
need an IP address in order for various voice and messaging services
to be provided, and for direct communications between devices in the
same or similar mobile networks.  The huge number of such mobile
devices precludes giving each an IPv4 global unicast address, and for
many purposes (but not global IPv4 connectivity) an IPv6 address may
work fine.

With a sufficiently large number of cellphones etc. on IPv6 like
this, it would be profitable to deploy many commercial IPv6 servers
to provide "content" for these devices would also be.  This would
make non-mobile IPv6-only services more attractive.  This "mobile
first" scenario for widespread IPv6 adoption is the most likely one I
can imagine.  A less likely scenario is certain countries providing
IPv6 only DSL etc. services to customers who cannot access any IPv4
service.

Hi Tom,

I will respond more fully to your thoughtful message separately,
probably with a subject like "Breaking IPv4's constraints".

The main body of text above is intended to be a statement of fact
about the major constraints which arise from our reliance on
widespread voluntary adoption.  This is to complement statements of
the functional goals of the solution - such as to provide a new form
of address space for end-user networks which scalably provides
multihoming, inbound TE and portability.  (Also, I think, support for
global mobility.)

Perhaps if enough people like this text - or some improved version of
it - we could add it to:

  http://tools.ietf.org/html/draft-irtf-rrg-recommendation

I am trying to keep it brief, but would like some kind of footnote to
at least acknowledge the inherently IPv4-centric nature of these
constraints - since IPv4 is not mentioned specifically in the main text.

While this text embodies what I always meant by "incremental
deployability" I am not trying to define this term.  You wrote
several times about my text as if it was to become "RRG "incremental
deployability" recommendations".

I want it to be a concise but reasonably complete statement of some
constraints which most of us agree are are real-world limitations we
must work within.

Only by implication is it a "recommendation" - for how to evaluate
potential solutions: proposals, architectures, candidate designs or
whatever.

I sense you are very concerned about the difficulty of establishing
Internet services beyond the finite constraints of IPv4.  I think
there's nothing which could be written now which would contribute to
a solution to this problem.  I think a footnote is the best way of
acknowledging this.

  - Robin

---

From: Tom Vest <[email-masked]>
Date: Fri, 17 Apr 2009 13:03:24 -0400

>    address space between ISPs.

Thanks once again Robin.

My concerns would be fully covered if the above language were changed  
slightly...

    A successful solution to the routing and addressing
    scaling problem is tightly constrained by the need
    for it to be both accessible to, and voluntarily adopted by
    the great majority of established and aspiring end-user
    networks who want or need multihoming, inbound
    traffic engineering and/or portability of their address
    space between ISPs. Because this voluntary adoption may
    occur over a period of years, the benefits of the solution
    itself should be broadly accessible to all end-user networks,
    regardless of their size or legacy (pre-solution) protocol
    endowments.

     1 - The solution must provide immediate and
         compelling benefits to any new or existing
	end-user network that adopts it, irrespective
	of how many other networks have so far adopted it.

>    foreseeable future, the problem only manifests in

I would prefer something along the lines of:

    Our goal is to solve the routing scaling problem in
    in a manner that will accommodate growth of both IPv4
    and IPv6 Internets. Because of the looming exhaustion of
    unallocated IPv4 addresses, this dual protocol orientation
    represents the narrowest possible scope that any candidate
    solution must address in order to have any chance of being
    successful.

>    which are expected to be mobile devices.                 *

*****

> same or similar mobile networks.

It's not an unimaginable scenario, but if China was truly hampered by  
a shortage of public IP addresses, that problem would have manifested  
itself years ago, in the course of extending non-mobile, IPv4-based  
Internet access. For a variety of reasons, China instead chose to  
develop alternative approaches for solving this problem -- mechanisms  
that are now coming to be known more widely as "carrier grade NATs."  
Scarcity of IPv4 was almost certainly not the primary factor driving  
this choice, but I would say that among the less obvious  
considerations, the commercial/competitive advantages of operator- 
imposed addressing were at least if not more important than more  
widely remarked-upon concerns about domestic order and state security.  
Thus, I would suggest that the China case tends to point to a quite  
different future than the one you suggest -- one that rather usefully  
illuminates the kind of concerns that I've tried to articulate in this  
exchange.

(for the record: I had the unique privilege of playing a leading role  
in the design, deployment, activation, and shortly thereafter the  
dismantling of the only international joint venture Internet access  
service in China -- the ill-fated "Legend-AOL"... an absolute  
commercial disaster, but an amazing/unique learning experience for me  
personally).

>  The huge number of such mobile

I agree with you that it's the most likely of all the plausible  
scenarios that one might imagine today.
Unfortunately, I think that it too is highly unlikely.

Not sure if it was your intent to incorporate the mobile scenario in  
the footnote, but I would recommend leaving it out.

> probably with a subject like "Breaking IPv4's constraints".

I'll look forward to that...

> deployability" recommendations".

I appreciate the clarification -- was not really clear about the  
"normative" content of the deployability text, apart from the obvious  
(i.e., no coercive authority, etc. -- already covered). In any case,  
any apparent mischaracterizations were the product of uncertainty, and  
not of any positive assumptions on my part that are inconsistent with  
your response... i.e., please consider them to be idiosyncrasies of  
phrasing with no particular relevance.

> Internet services beyond the finite constraints of IPv4.

That is fair to say!

> I think there's nothing which could be written now which would  
> contribute to a solution to this problem.

I would hesitate to reach that conclusion, in part for the following  
reason(s):

1. Any future RRG "final product" is almost certain to come after IPv4  
exhaustion and privatization (de facto and/or de jure).

2. If, after that point, RRG solutions are primarily or exclusively  
oriented toward IPv4 scaling/perpetuation, that will effectively put  
RRG in the novel role of maintainer of private/proprietary resources  
-- "proprietary" in a sense not unlike that in which specific branded  
commercial hardware or software is proprietary. The owners of the new  
proprietary resources may be numerous and widely distributed, but it  
will forevermore be a closed group, with something like "membership by  
invitation only."

3. Over the past decade or so, the population of routing system  
participants has been very dynamic, with appx. 5-10% growth in the  
form of new entrants each year (based on RIR "initial allocation"  
transactions involving IPv4) and a somewhat lower (and much harder to  
estimate) rate of departures due to M&A and/or termination of  
service.* After IPv4 runout or privatization (whichever comes first),  
the new add rate is likely to slow down to a trickle, leaving an ever- 
growing population of frustrated would-have-been routing system  
participants. At the same time, the drop rate is likely to increase as  
some large, well-heeled operators accelerate their pursuit of low- 
hanging IPv4 acquisition targets. Overall, this change is going to  
represent a radical, unprecedented break with the Internet growth  
processes of the last two decades. I don't think that it will take  
long for the general public to notice, and react.

If and when all of this comes to pass, "open addressing" will quickly  
achieve, at least, the same level salience/urgency as "scalable  
routing" in the overall RRG problem space. In anticipation of (at the  
very least) this possibility, I would suggest that it would be prudent  
for RRG to capitalize on every possible opportunity to reaffirm its  
commitment to delivering a solution to both problems, starting now.

Thanks again for being receptive to my troublesome and somewhat  
belated expressions of concern...

Tom

*(In the US at least, these routing system population dynamics almost  
exactly parallel those of the commercial banking sector).

---

From: Robin Whittle <[email-masked]>
Date: Mon, 20 Apr 2009 01:26:12 +1000

Short version:  Long reply to Tom Vest, discussing his concerns
                about IPv4, even with a scalable routing solution,
                hitting the limits of its address range and so
                causing difficulties for aspiring new ISPs.

                The same issues are covered more concisely in the
                footnote to the draft list of constraints.

                Also, some discussion of the TTR Mobility approach
                in the hope that people will read it and so not be
                so alarmed at the mention of mobility with scalable
                routing.  I first described this in June 2007 and
                this full description, with diagrams, has been
                available for 8 months:

                  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

Hi Tom,

The list of constraints due to the need for voluntary adoption is at:

   http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

I have added a second last paragraph which mentions some of your
concerns.  As I discuss here and in the Footnote at the above page, I
think your concerns are important but that there is no need to change
the list itself.  I intend it to be a factual list of constraints we
can all agree to - even if some of them seem to be at odds with a
truly satisfactory outcome in terms of endless address space and the
avoidance of barriers to new entrants.

Here is a reply to your message in the "Constraints on a solution
..." thread.  Thanks again for your thoughtful discussion and

>    regardless of their size or legacy (pre-solution) protocol

I have amended the text to:

   be both accessible to, and voluntarily adopted by

and I hope you will find that the new second-last paragraph at least
partially mentions your concerns.

   The benefits resulting from the best imaginable
   solution for IPv4 may be limited by shortage of IPv4
   space and resulting difficulties for new networks in
   gaining this space.  While no such restrictions exist
   with IPv6, constraints similar to 2, 3, 4 and 5 are
   likely to continue to be a barrier to widespread
   migration to IPv6.

I want the list of constraints to be a terse description of things we
all (ideally) agree are true.  The focus is on what is needed for
end-user networks and their ISPs to adopt the solution.

As a statement of fact, this text does not assume that there is a
solution which will meet all these constraints.  I think there will
be solutions which meet them all in IPv4 - but I agree that during '
adoption and in the decades to come, no matter how ubiquitously the
scalable routing solution is adopted and how well it works, the
address space shortage is likely to limit the IPv4 Internet's
capacity to provide multihoming etc. for all end-user networks which
want it.  This includes especially those end-user networks which are
to be established in the future and/or those whose Internet access
would benefit from newly established ISPs who, in practice, may find
it difficult to gain the address space they need.

I added another paragraph at the end:

   The above constraints generally apply for IPv4 when
   end-user networks already have IPv4 space and need to
   retain full connectivity to all, or almost all,
   existing hosts.  They may not apply in other
   circumstances, such as if IPv4 space is unobtainable
   and/or if the functionality of devices in a newly
   established end-user network does not depend so
   strongly on connectivity to IPv4 hosts and/or if the
   hosts in the new network are all purpose built for
   it.  Development of a large IPv6-based cellphone
   network may exemplify these circumstances.

which summarises the example I give in the Footnote about how I can
imagine a newly established IPv6-based cell-phone network not being
so affected by all these constraints.

>         compelling benefits to any new or existing        <<

Thanks, I have used this suggestion.

Regarding the Footnote, which I have completely rewritten, you

>>    foreseeable future, the problem only manifests in

Similar text is in the new Footnote.

>    solution must address in order to have any chance of being

I don't agree with this, for a number of reasons.

Firstly, I don't assume that the only alternative to IPv4 is IPv6.

Secondly I don't assume there is a solution to the IPv4 address
shortage which enables most users to transition to some new protocol
without disruption and costs they would in fact accept.  I think most
end-users would not accept having to abandon applications which work
fine on IPv4 or having to use a network which did not give them full
access to all hosts, most of which are on IPv4.

Can you propose a solution which achieves the goals you want to set
for any scalable routing solution?

I believe the routing scalability problem can be solved for IPv4 and
IPv6 separately, but I am yet to see how any system, involving
scalable routing or not, could provide for mass adoption of IPv6 or
some other protocol or disruption-free migration to that from IPv4.

I think you have argued that a scalable routing solution for IPv4
will not succeed fully if at some point in the future, there is no
more address space to accommodate new end-user networks.  My counter
argument is that in that case, those end-user networks wouldn't be
able to get address space they would use in an unscalable way - so
the real problem is address shortage and not a deficiency in routing
scalability.

I think the IPv4 address shortage - in absolute terms and/or in terms
of overly concentrated control of usable space in the hands of
established networks - will be the biggest limitation on the
successful development of the Internet even if the scalable routing
problem is fixed.

I think IPv4 address exhaustion and fears of (to new entrants) unfair
address distribution outcomes is a far better understood problem than
the routing scaling problem.

Ivip is well suited to making the very best use of IPv4 space - for
instance by giving networks 1, 2, 3, 4, 5 or whatever number of
contiguous IPv4 addresses they need.  I think the most numerous types
of end-user networks will be those which use one or a few IPv4
addresses.  So theoretically we could have a billion such
networks occupying a billion or two addresses with close to 100%
utilization efficiency.

I think LISP or APT could also be used in much the same way, although
their EIDs are restricted to prefixes of 1, 2, 4, 8 etc. IPv4
addresses.  The designers of neither proposal seem to be very
interested in networks with a handful of addresses.  I think the LISP
designers in particular are heavily focused on substantial end-user
networks which have 256 and more IPv4 addresses.

Yet the whole benefit of LISP-ALT is its ability to handle
essentially limitless numbers of EIDs and therefore end-user networks.

I think LISP-NERD, the fastest and simplest of the proposals, could
easily scale to 10 or 20 million EIDs.  Yet it is impossible for me
to imagine more than this number of fixed (non-mobile) end-user
networks ever needing multihoming.  With a population of 10B people,
10M end-user networks would be one for every 1000 people.  I just
cant imagine any more than this number of fixed networks having such
a desire for multihoming to get two physically separate links and
ISPs.  Nor can I imagine more than this number of fixed end-user
networks needing portability.

So if LISP-ALT is going to be better than LISP-NERD, it is only going
to be because LISP-NERD can't scale to the number of EIDs which
LISP-ALT can handle with ease.  (LISP-NERD, APT and Ivip, is far
superior to LISP-ALT or TRRP because these first three involve little
or no delay to any packets while the latter two involve often
significant delays to initial packets due to the delays and perhaps
packet losses inherent in their completely decentralised global query
server networks.

My point is that if LISP-ALT is to be better than LISP-NERD, then it
must be due to LISP-ALT supporting mobile end-user networks.

It can do this, with the TTR Mobility approach - as long as the
designers accept that this involves the ETR is separate from the
end-user network (they currently assume it within the end-user
network) and if they accept the ETR not even being in the access
network - it is in a network of the TTR company (or some network of
an ISP which is where the TTR company has its TTR).  The TTR is
typically physically close to the access network, such as 100km or less:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

So if LISP-ALT is to be superior to LISP-NERD for IPv4 - meaning it
is going to support 100M, 1B, 2B etc. EIDs (and nearly the same
number of end-user networks), then it is going to have to support
Mobility - and mobile devices probably only need a single IPv4 address.

The same is true of APT and Ivip - both can use the TTR Mobility
approach and both must support mobility if they are going to cope
with more end-user networks than LISP-NERD could comfortably handle.

Ivip is intended to scale well to billions of micronets (EIDs in LISP
terminology) and to have many advantages over LISP-ALT/NERD and the
other alternatives, including better modularity and better support
for the TTR Mobility model.

To summarise the above:  Ivip should be highly capable of using IPv4
space in a highly efficient manner.  While the designers of APT and
LISP don't seem to be thinking in these terms, I think their systems
could do much the same.

So I think the best scalable routing system for IPv4 will enable
a very high utilization of IPv4 space.  In principle, this may
provide considerable leeway for new-entrant end-user networks.

With Ivip the new SPI space (Scalable PI space) for each end-user
network will be rented from some company which "own" a much larger
"Mapped Address Block" (MAB), which is a BGP advertised prefix.  A
typical MAB may contain 64K to 1M or whatever IPv4 addresses - and
this can be used for micronets of all sizes, including in principle
64K or 1M or whatever single IPv4 address micronets.

The companies which "own" the MABs and rent out the space will not
necessarily be ISPs.  They will generally be commercial operations
like ISPs, but they don't need an access network.  The MAB company
needs to ensure there are OITRDs (Open ITRs in the DFZ - like LISP
Proxy Tunnel Routers) for its MABs, ideally multiple OITRDs
physically distributed all over the Net.  If it runs them itself, the
MAB company needs to be an AS.  However, it could easily pay some
other company to run those OITRDs.

In a fully developed Ivip system, the global unicast address space
would be divided into several types of usage.  The actual BGP
advertised prefixes for these could be all over the place.

 1 - Conventional prefixes for ISPs which use it for their own
     internal purposes, their servers and routers etc. and for
     providing PA space to many of their customers.  This PA
     space includes substantial end-user networks which need
     multiple addresses.  It also includes the hundreds of
     millions of DSL, cable modem etc. customers who currently
     use a single IPv4 address each.

 2 - Conventional prefixes for end-user networks which are ASes
     and are not using Ivip-mapped space.

 3 - MABs which are "owned" by MAB companies and rented out to
     end-user networks which are using this Ivip-mapped SPI space.

If we imagine an idealised situation in which all the end-user
networks adopt SPI space then this leaves only three categories of
address space:

 1a - ISPs use this for their internal purposes, and for ITR and
      ETR addresses.

 1b - That part of ISP's prefixes which are devoted to their
      numerous PA customers.

 3 -  MAB space which is used with potentially very high utilization
      rates approaching 100% for SPI-using end-user networks.

In the long-term future, here are some figures:

 1a -   100M   } ISPs
               }
 1b - 2,000M   }

 3 -  1,600M   } MAB companies, which are not necessarily ISPs.

Let's say there are 10M fixed end-user networks using SPI space and
that on average they need 80 IPv4 addresses.  Most need less, such as
4 or 8 and quite a few need only 1 or 2.  However a small number of
these are corporations, universities, hospitals, government
departments, hosting companies etc. which need thousands or hundreds
of thousands of IPv4 addresses.

This leaves 800M IPv4 SPI addresses available for mobile end-user
networks - physically mobile devices such as cellphones, media
players or PCs.  There won't be much difference in the near future -
an Iphone or Gphone is arguably all these already.

These will work fine with a single IPv4 address.  These devices can
use this address via Ivip and the TTR Mobility system no matter what
their connection to the Net, including behind NAT.

Many such mobile devices will have their CoA on one of the 1b
addresses - they will have a PA address to themselves.  Others will
be behind NAT, so multiple such devices might be on WiFi etc. behind
NAT and sharing a single 1b address.

These mobile devices need at least one CoA - and they need their own
unique SPI address - type 3, which they rent from the MAB company,
which is not necessarily an ISP.

I think that in your model of IPv4 adoption, the crucial question is
how new entrants:

  Aspiring ISPs

  End-user networks which want an AS and their own PI prefixes.

gain access to address space when it is all held by either existing
ISPs or existing PI-using end-user networks.

In the above model of Ivip usage (I think LISP or APT would be the
same in principle) the system does provide SPI space for potentially
far more end-user networks than could ever have the resources to get
their own PI space.  1, 2, 3, 64 to 128 or similar smallish number of
IPv4 addresses will surely be less expensive than the full costs of
gaining one or more /24 prefixes and having ISPs advertise it in BGP.

(Theoretically the current /24 limit on the prefixes routers accept
could be loosened to allow 128, 64, 32 etc. address prefixes - but I
can't imagine this happening as long as there is already
unsustainable growth in the number of DFZ routes.)

Mobile end-users are dependent on ISPs in one way or another to give
them Internet access and a CoA.  They are also dependent on a MAB
company from whom they rent probably a single IPv4 address, which
they can use via any ISP in the world.  They also need to pay a TTR
company, which needs its own 1a type of space to run its TTRs (which
are ETRs as far as the Ivip system is concerned).  However a single
TTR on a single IP address can simultaneously handle dozens to
hundreds of simultaneous mobile end-users via the one IP address.

A non-mobile end-user network does not need any PA space - since it
has no CoA.  They need one or two ISPs to provide Internet access.
No matter how many SPI addresses each end-user network has, these
only require a single ETR in each of its ISPs.  Each such ETR can
simultaneously serve the needs of many such end-user networks.

With Ivip, the ETRs for fixed end-user networks are in the ISPs they
use.  This is totally different than for LISP, in which each end-user
network has its own ETR for each ISP.  So for LISP, no matter how
many EID addressees the end-user network has, it consumes at least
one IPv4 address for the ETR it uses for each of its two or more
ISPs.   These ETR addresses come from the ISP's 1a type space.  So an
ISP with 10,000 LISP end-user networks needs at least 10,000 separate
type 1a IP addresses for all these ETR addresses.

With Ivip, the same ISP might handle 100 end-user networks with each
of its ETRs, so it only needs 100 type 1a IPv4 addresses.

In this model, it is vital that aspiring ISPs AND MAB companies can
get the address space they need.  To the extent that they can and to
the extent that the compete and provide good services, this enables
the actual SPI-using end-user networks to get the Internet access
they need AND the SPI address space they need.

So your concern about IPv4 address space shortage and control by
incumbents is just as valid as without Ivip.  The details are
different, because MAB companies are not necessarily ISPs, with all
their physical infrastructure in particular locations.

I can imagine Ivip working nicely and providing really high
utilization for the IPv4 address space - provided it is made
available by whoever currently "owns" it.  If, as I suspect, we will
be stuck in IPv4 for a very long time, I think there will be
increasing regulatory pressures to free up underutilised address
space to make it available to ISPs and MAB companies which will use
it efficiently.  Also, those companies which do use it efficiently
will be able to derive profits and so pay a price for the space which
makes it attractive for underutilising companies to sell it.

Still, depending on the world's population and how important it will
be for cellphones/PCs to have a globally mobile IPv4 address, there
may well be a point, even with the fairest distribution of the 3.7B
global unicast IPv4 addresses, where the demand can only be met by
putting some or many mobile devices on IPv6 and/or by forcing some or
many DSL etc. customers onto some second-rate kind of NATed service,
rather than the current practice of giving each customer their own
global unicast IPv4 address.

This is a long way of saying that in principle I agree with your
concerns, but that I can imagine technical and commercial
developments - such as Ivip with TTR mobility - which enable the IPv4
address space to be utilized close to 100%.

Since I foresee the barriers to mass IPv6 adoption (except perhaps
for cellphones) will remain very high, I think it will generally be
cheaper and easier at any one point in time to keep making the most
of IPv4 than to for anyone to try to migrate to IPv6 without using an
IPv4 address.

Arguably, this would be a painful, drawn-out torture.  Maybe it would
be best if the entire IPv4 Internet evaporated sometime soon and
forced everyone to move to IPv6 instead.

It is not a popular position, but I think there is a lot of scope for
squeezing more and more from IPv4 and that we will probably be
depending on it as THE global Internet for decades to come.

At the same time, I am keen to facilitate any development which could
provide a better alternative.

> It's not an unimaginable scenario, but if China was truly hampered by a
> shortage of public IP addresses, that problem would have manifested
> this choice, but I would say that among the less obvious considerations,
> the commercial/competitive advantages of operator-imposed addressing
> were at least if not more important than more widely remarked-upon
> concerns about domestic order and state security. Thus, I would suggest
> that the China case tends to point to a quite different future than the
> one you suggest -- one that rather usefully illuminates the kind of
> concerns that I've tried to articulate in this exchange.

I understand from this that what I wrote about widespread IPv6
adoption scenarios is less likely to occur than I suggested.

I am keen to know of anything concerning IPv6 adoption or the lack
thereof in large mobile networks, in China and in other countries
where one might expect the IPv4 address shortage to strongly motivate
the adoption of IPv6-only or perhaps dual stack services.  I think it
is relevant to scalable routing, but perhaps it is too contentious to
be put on the RRG list.

I see you have some research materials at:

which leads to a page at your site http://www.eyeconomics.com .

> (for the record: I had the unique privilege of playing a leading role in
> the design, deployment, activation, and shortly thereafter the
> service in China -- the ill-fated "Legend-AOL"... an absolute commercial
> disaster, but an amazing/unique learning experience for me personally).

Very interesting.  I hope this learning experience somehow pays the
bills today!  I see http://www.caida.org/home/staff/tvest/ has your
your extensive ~2006 CV and I understand you are part of the RIPE NCC
Science Group, though I found no web pages for this.

> Unfortunately, I think that it too is highly unlikely.

OK - but see the footnote at:

 http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/

for why I think the total functionality of a large IPv6 cellphone
network wouldn't depend so much on IPv4 connectivity as would
conventional Internet services.

> Not sure if it was your intent to incorporate the mobile scenario in the
> footnote, but I would recommend leaving it out.

I wrote a longer footnote which mentions it extensively.

I think that for most RRG folks:

   Scalable_Routing + Mobility = Teeth-itch

I hope people will read:

   http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

and so hopefully realise that a core-edge separation scheme such as
LISP, APT or Ivip can directly support a new ETR-like Translating
Tunnel Router approach to global IP address mobility, for IPv4 and
IPv6.  This would be a good complement to existing MIP techniques.
It does NOT involve frequent mapping changes - such as every time the
MN gets a new CoA.

> I'll look forward to that...

I am hoping that this message and the footnote is a good response to
the interesting matters you have raised so far.

>> ...
> (i.e., no coercive authority, etc. -- already covered). In any case, any
> apparent mischaracterizations were the product of uncertainty, and not
> of any positive assumptions on my part that are inconsistent with your
> response... i.e., please consider them to be idiosyncrasies of phrasing
> with no particular relevance.

OK.

>> ...

With good reason, I believe.  While I can imagine Ivip and fair
distribution of IPv4 space creating a better IPv4 outcome than most
people think possible, even if these things eventuate as I hope,
there will be many years of difficulties beforehand.

> exhaustion and privatization (de facto and/or de jure).

I agree.

> oriented toward IPv4 scaling/perpetuation, that will effectively put RRG
> in the novel role of maintainer of private/proprietary resources --
> "proprietary" in a sense not unlike that in which specific branded

Yes - but this assumes that the RRG solution could have done more to
enable widespread adoption of IPv6.  That is outside the formal scope
of the RRG's goals, AFAIK.  If I can make Ivip in some way facilitate
migration from IPv4 to IPv6 or any better alternative, I will do so.

Without any specific suggestion for how a scalable routing solution
or anything else could help with easing the dependence on IPv4, I
don't think it is fair to insist that the RRG or anyone else should
do more to achieve such a goal.  I count it as a highly desirable
goal, but not a requirement.   If I thought there was a way of doing
it, then I would change my mind and regard it as a requirement -
because it is so important.

I am trying to keep the list of constraints and its associated test
as terse as possible.  The first of the new paragraphs at least
acknowledges that in practical terms, the constraints apply entirely
within IPv4 and also suggest that constraints similar to 2, 3, 4 and
5 are a major factor in the failure of IPv6 to be widely adopted.

Please suggest improvements, but I am really trying to keep it short.
 It is my experience that anything longer than about 4 paragraphs
tends to be largely ignored by some key RRG people.

> participants has been very dynamic, with appx. 5-10% growth in the form
> of new entrants each year (based on RIR "initial allocation"
> estimate) rate of departures due to M&A and/or termination of service.*
> After IPv4 runout or privatization (whichever comes first), the new add
> rate is likely to slow down to a trickle, leaving an ever-growing
> population of frustrated would-have-been routing system participants. 

I know little about these things, but it is easy for me to imagine
the big fish eating the small fish - especially with the economic
crisis.

I guess the most numerous and significant of these aspiring new
networks - in terms of competition policy and in terms of development
of Internet services in countries which are not well served at
present - would be new ISPs and cell-phone companies wanting to be
ISPs to give their customers direct Internet access.

There would also be a growing number of end-user networks - new and
existing - who have the resources and the impetus to get multihoming,
TE and portability with existing BGP techniques, if they could only
get their own PI space.

I think a good scalable routing solution will enable many such
end-user networks to use the new kind of address space, without the
investment in BGP routers and expertise.  Furthermore, many more
smaller networks who would never have the resources or justification
to get PI prefixes will be able to use the new kind of scalable
address space too, for low costs and by using as few IP addresses as
they really need.

> At the same time, the drop rate is likely to increase as some large,
> well-heeled operators accelerate their pursuit of low-hanging IPv4
> acquisition targets. Overall, this change is going to represent a
> radical, unprecedented break with the Internet growth processes of the
> last two decades. I don't think that it will take long for the general
> public to notice, and react.
> 
> achieve, at least, the same level salience/urgency as "scalable routing"
> in the overall RRG problem space. In anticipation of (at the very least)
> this possibility, I would suggest that it would be prudent for RRG to
> capitalize on every possible opportunity to reaffirm its commitment to
> delivering a solution to both problems, starting now.

I fully agree that the RRG should discuss and encourage the design of
scalable routing solutions which also help with:

 1 - Improving IPv4 address space utilization.

        I think any decent core-edge separation scheme will be
        capable of this.

 2 - Fairer allocation of IPv4 addresses to new ISPs.

        I can't see how the RRG can affect this, other than by
        helping to design a flexible, efficient, scalable
        routing system which enables finer slicing and dicing
        of the address space, which is the same thing as 1 above.

 3 - Adoption of and transition to IPv6.

        This nut has never come close to being cracked.  I think
        it may be impossible.  I don't assume it is so, but until
        you or anyone else has a scheme which will work, in part
        by meeting constraints similar to those I listed, I don't
        think it is fair to expect anyone else to solve this
        infernal problem.

I also believe that because the TTR mobility approach is so desirable
and is such an easy extension of a core-edge separation scheme, that
the RRG should take a greater interest in it.

Also, mobility is the only way I can imagine that any of the
proposals LISP-ALT, APT, Ivip or TRRP will be required to scale
beyond the 10M or so EIDs which could easily be handled by the
simpler, more direct and optimally fast LISP-NERD.

> Thanks again for being receptive to my troublesome and somewhat belated
> expressions of concern...
> 
> Tom

I really appreciate your messages.  This field has many technical,
human and commercial complexities.  I think progress often results
from detailed, thoughtful discussions.

  - Robin

---

From: Tom Vest <[email-masked]>
Date: Sun, 19 Apr 2009 19:30:58 -0400

Hi Robin,

Thanks for taking (what must have been considerable) time to reply in  
detail.
The above two modifications do, indeed, adequately reflect the  
concerns I expressed.

> want it.

Of course I agree on this point, but this particular goal was never  
explicitly stated, nor implied, nor even contemplated by me.
The idea that merely "wanting it" -- i.e., full access to all of the  
'rights, privileges, and responsibilities' that are enjoyed by direct  
participants in the Internet's routing subsystem -- is or should be by  
itself sufficient for any institution to lay claim to a share of that  
system's finite carrying capacity -- fundamentally contradicts the  
findings of my own personal experience and research (maybe common  
sense too, if I presumed to know what that was).

More precisely stated, the implications of my recent work suggests  
that the very nature of the constraints that dictate the requirement  
for "incremental deployability" are *the same* structural constraints  
that absolutely dictate that any/every conceivable current and future  
routing system that could ever be adopted and deployed in this manner  
will -- inevitably, necessarily -- embody some set of features that  
ultimately make all such systems vulnerable to some version of *the  
same* systemic risks to which the current system is vulnerable, namely:

-- "inflation" (c.f., generalized/cumulative/direct impact of  
individual demands on routing system carrying capacity that are borne  
by all/other participants in that routing system)

-- "deflation" (c.f., generalized/cumulative impact of non- 
availability of routing system capacity and/or address resources,  
which manifests as diminished growth potential for existing   
participants and a progressive barrier to the emergence of potential  
new participants)

-- "stagnation" (c.f., generalized/cumulative impact of non- 
availability of "standard"/"DFZ" routing system capacity and/or  
"standard"/uniform address resources, which manifests as a progressive  
degradation of and loss of confidence in the consistency/ 
predictability of precisely those "network effects" that make a  
shared, inter-domain routing system so valuable to its direct  
participants, as well as to indirect stakeholders).

Another way of putting this is that the specific problems that RRG is  
trying to solve are actually just specific manifestations of more  
general risks -- risks that are *not* primarily or exclusively an  
artifact of current routing and addressing technology. Rather, these  
risks are a natural, organic, and (probably) largely unavoidable  
feature of *all* liquidity mechanisms -- i.e., all "technically  
mediated exchange systems." They are unavoidable because a necessary  
condition of possibility for the existence/emergence of such mediated  
exchange systems is the prior existence of a population of diverse,  
independent, often competing economic agents (individuals,  
institutions, etc.), members of which might often enjoy opportunities  
to productively interact with one another, but for the existence of  
various "transactional frictions" (foregoing details on this for now).  
"Liquidity system" and/or "technical medium of exchange" are the  
interchangeable terms that I've adopted [1] to describe the thing that  
emerges in some (not all) situations, which is functionally  
distinguished by its capacity to reduce if not eliminate those  
frictions for all participants who voluntarily adopt it -- thereby  
becoming participating members in that particular liquidity system.  
The intrinsic friction-reducing benefits of participation (growth,  
profit, etc.) attracts and retains participants within that system,  
which creates/strengthens incentives for system participants to  
interact with each other ever more intensively (producing as a result  
the devision of labor, specialization, etc.). However, those same  
benefits also have the organic effect of intensifying competition  
between system participants, which eventually results in their  
becoming aware of the liquidity mechanism itself, and its possible  
uses as a strategic, (anti)competitive tool.

I'll spare the list the rest of the evolving story. Those interested  
can find more of it is on my personal website (links below).

In any event, I wholeheartedly share RRG's ambition to contribute to  
the alleviation these problems. The story above explains why systems  
like ours need liquidity mechanisms, and what benefits they confer --  
and risks they impose -- on those systems that are lucky enough to  
have one. However, that "need" does not entail that it will be  
fulfilled. Obviously, another absolute condition of possibility is the  
existence of some candidate mechanism(s) that is capable of initiating  
and sustaining this liquidity process over time. Not all candidate  
liquidity mechanisms are created equal, and I think of RRG's core  
mandate as the definition of the "best possible" successor liquidity  
mechanism for the Internet that we have inherited -- with "best  
possible" defined as (max. [probability of adoption * probability of  
sustainability, given known structural and exogenous risks]). That  
said, I do not believe that the problems that RRG is trying to solve  
are amenable to any complete or final "solution" -- nor even to much  
symptomatic relief unless the root cause(s) of the problems are  
recognized and addressed directly.

And that brings me back to the point of departure, Robin, which was  
the contrast between "desire" or (self-asserted) "need" vs.  
*eligibility*. It seems to me that the formula that made it possible  
for the last RRG-relevant domain-wide solution (CIDR) to "work" --  
i.e., to be both (1) accepted by, at least, the minimum set of parties  
required to get started, and (2) to continue to be regarded as  
tolerable/acceptable by the majority of direct, indirect, and aspiring  
stakeholders thereafter -- involved three key ingredients:

1. The idea that there is a (i.e., at least one) technically  
justifiable/defensible set of criteria that can be used to determine  
an institution's "eligibility" to consume some share of the common/ 
default routing system's finite carrying capacity. To maximize  
sustainability over time, these criteria should establish a link  
between individual-level demand for participation and the potential  
system-wide benefits of allocating a share of finite system carrying  
capacity to the aspiring participant.

2. To further maximize sustainability over time, eligibility criteria  
should be responsive to changing conditions, e.g., an expansion of the  
system's finite carrying capacity should be reflected in the  
corresponding eligibility criteria. That makes eligibility criteria  
management dependent on the active participation of those who are  
closest to the implementation technologies and practices.

3. These eligibility criteria should, in both principle and practice,  
form the primary basis for allocating finite routing system carrying  
capacity. Achieving this in a sustainable way is only possible if the  
eligibility criteria are administered, and the associated resources  
distributed, through some neutral, transparent mechanism that is  
substantially insulated from the competitive dynamics that govern most  
direct interactions between existing liquidity system participants. In  
other words, the allocation mechanism must be insulated from the very  
dynamics that make the liquidity system itself so valuable, and which  
make the potential risks and consequences of liquidity system capture,  
collapse, or premature obsolescence so severe.

So, to your point about not being able to "provide multihoming etc.  
for all end-user networks which want it," I respond that that is not a  
realistic goal, given the joint constraints of technology and human  
nature. I seriously doubt that it ever will be, because in an  
uncertain world full of real and/or potential competitors, people will  
always "want" more than they currently "need," and nothing short of  
infinite carrying capacity or a change in human nature can remedy  
that. However, I believe with even greater seriousness that we must  
aspire to preserve -- or barring that possibility to recreate with the  
shortest humanly possible interruption -- those three conditions, and  
the kind of "eligibility-based" system admission process that results  
when those conditions are in place.

I believe this strongly because the history of liquidity systems past  
suggests that if those conditions are not sustained or restored in  
(effectively no) time, whatever endures may not be redeemable, or  
worth redeeming for that matter, and in any event system maintenance  
by then is then likely to have become someone else's problem. Just as  
the "need" for a liquidity mechanism like TCP/IP does not guarantee  
that one will emerge and be widely adopted, neither does that need  
guarantee that an existing mechanism will endure, or in enduring  
superficially ("in form only") continue to provide those substantive  
benefits that make liquidity systems so valuable, and so vanishingly  
rare in the wild.

Monetary liquidity systems have come and gone in the past -- but the  
ones that have both "gone" and subsequently "come back" have almost  
universally been backed by sovereign power -- and 100% of those were/ 
are exclusively national in scope. What that portends for the future  
of our own, first-of-its kind, extra-territorial, attachment-based  
liquidity mechanism is left as a exercise for the reader.

Note: To prevent the weight of this message from giving the Internet a  
hernia, I've split out comments on the footnote next in a separate  
message, to follow shortly.

Thanks again for your responsiveness...

TV

[1] The terms come a newish subfield of academic/operational  
macroeconomic research that focuses on topics like the origin of  
"money" and the sustainable management of monetary systems, most of  
whose leading contributors seem to be current or former commercial  
banking professionals or central bank/monetary policy decision makers.

Definition and comparison of "technical media of exchange" aka  
"liquidity mechanisms":

http://www.eyeconomics.com/eyecononomics.com/MoEDdefined.html

Expected consequences of the looming transition in Internet "liquidity  
mechanisms":

http://www.eyeconomics.com/eyecononomics.com/NetStagflationPaper.html

---

From: Robin Whittle <[email-masked]>
Date: Mon, 20 Apr 2009 19:02:04 +1000

Long message ahead:

                Discussing Tom's "Liquidity Systems" view of the
                Internet's routing and addressing system.

                I haven't yet found any insights from it which
                help with scalable routing.

                Discussion of how with LISP, APT, Ivip or TRRP
                more and more address space will be needed by the
                "MAB" companies who rent the space out as EID
                prefixes micronets or whatever is the name for the
                new kind of scalable address space for end-user
                networks.

                It follows that a greater proportion of space will
                be needed by these companies, and a lesser proportion
                by ISPs.  I give a contrived example of an ISP with a
                single /24 being able to serve 10,000 end-user
                networks which between them have 100,000 IPv4
                addresses of EID / micronet etc. space.

                Also, how competition between these MAB companies
                will help drive a healthy market for obtaining
                space which is less profitably utilised than will
                be possible with a good scalable routing solution.

Hi Tom,

Thanks for this:

> The above two modifications do, indeed, adequately reflect the concerns
> I expressed.

It did take a while to work it out.  Originally I wasn't going to add
anything, but I am happy your concerns and discussion prompted me to
add these two paragraphs.

> findings of my own personal experience and research (maybe common sense
> too, if I presumed to know what that was).

With Ivip at least, and I think also for LISP, APT and TRRP, wanting
/ needing (who is to say which is true?) multihoming, TE and
portability and then getting it via Ivip certainly does not involve:

    full access to all of the 'rights, privileges, and
    responsibilities' that are enjoyed by direct participants
    in the Internet's routing subsystem

If a network either does not exist, or exists on PA space, then it
has a price to pay to obtain some Ivip-mapped "SPI" (Scalable PI)
address space, whether it be a single IPv4 address or thousands of
such addresses.

If the network already exists, it also need to be renumbered to the
new SPI address range.

There will be ongoing rental fees for the SPI space, paid to the MAB
company who "owns" or at least has rights to, the MAB (Mapped Address
Block) which is advertised in BGP and which contains the User Address
Blocks (UAB) of many other end-user networks.  How each end-user
network splits up the one or more IP addresses in its UAB into
micronets (1 or more contiguous IP addresses with a single mapping)
is up to the end-user network.  They will typically pay per mapping
change - to the MAB company, who will typically pay a Root Update
Authorisation System (RUAS) company to send the mapping change to all
the world's full database Query Servers.

The MAB company will also directly or via some other company run
OITRDs (Open ITRs in the DFZ) which are necessary for handling
packets sent to SPI addresses in its MABs from hosts in networks
without ITRs.  Since traffic volumes for OITRDs would vary
enormously, the end-user network will typically need to pay in
proportion to how much traffic these OITRDs handle for their micronets.

So merely "wanting" something won't make it happen.  The end-user
network needs to be willing to pay for all this.  It won't be very
expensive when compared to the total costs of getting an AS, PI space
etc.

While the end-user network gets its own set of global unicast IP
addresses (SPI addresses), which it can use at any ISP which has an
ETR, this doesn't to me constitute what you wrote and which I quoted
above.

These end-user networks don't need to know anything about BGP etc. or
be responsible for running DFZ routers.

This will be a mass-market service, competitively provided by
multiple MAB companies all round the world.  These MAB companies may
be ISPs - but they need not be, because there is no need for physical
infrastructure other than someone to run their OITRDs in various
locations around the Net.

Before Ivip etc. for an end-user network to gain global unicast
addresses it either had to rent some PA space from an ISP or get an
AS, and gain a PI prefix (and then have ISPs advertise this prefix).

Once Ivip or similar is available, they will have a third, highly
attractive, option: rent some SPI address space in a MAB and use it
at any ISP which has an ETR.

Imagine a scenario in which an ISP-M sets up shop purely for the
purpose of serving customers with SPI addresses.  ISP-M may or may
not have MABs of its own to rent space in.  I will assume it doesn't.
 Its customers will already have rented their SPI space from one of
many MAB companies.

In Ivip, the ETR is typically in the ISP network.  (Except for with
TTR mobility, where the ETR is the Translating Tunnel Router, ideally
located near the access network.)  Let's say each ETR can handle 100
end-user networks.  Let's say ISP-M has 10,000 separate end-user
network customers, each with an average of 10 IP4 addresses of SPI space.

The entire ISP-M could be run nicely on a /24.  It needs 100 global
unicast ordinary ("RLOC" in LISP terms) IP addresses for its ETRs.
It can use the same addresses for ITRs too - and ITRs will frequently
be in the end-user networks anyway, including in some sending hosts.
 ISP-M also needs addresses for query servers etc.  So ISP-M could be
serving 100,000 IP addresses of SPI space via 256 "RLOC" addresses.

If all those end-user networks were multihomed to another ISP, then
in theory they could all be served by two similar ISPs ISP-M and
ISP-N, each with only a /24 of space themselves.

LISP as currently conceived is a little differently.  The LISP
designers assume that the ETRs will be at the end-user network.  So
each such network needs a single "RLOC" address from each of its ISPs
for the ETR which links to each ISP.  Therefore, ISP-M would need at
least 10,000 RLOC addresses and so would ISP-N.  LISP-ALT, as
currently conceived, needs the ETRs to be at the end-user network for
at least two reasons.  Firstly, reachability of the ETRs from any ITR
is taken as meaning reachability of the end-user network.  (This is
not the case for Ivip.)  Secondly, the ETRs need to act as a unified
system, so they need to be owned and operated by the end-user
network, and robustly linked together no matter what outages etc. occur.

While LISP is less efficient with RLOC space than Ivip, the point
remains that an ISP serving SPI customers does not need to have much
space at all.  One or more end-user network with 64K addresses can in
principle work fine from a single ETR.

This significantly changes the address requirements for ISPs.  There
still is an address requirement for this SPI space - but it is
handled by MAB companies gaining rights to large slabs of address
space, and then making money by renting that space out.

For an end-user network in any one physical location, they may only
be able to choose between one or a small number of ISPs, since only
those ISPs have physical infrastructure to their site.  However, the
end-user network's SPI address can come from any MAB company in the
world.

More on the competition effects of this below.

> More precisely stated, the implications of my recent work suggests that
> the very nature of the constraints that dictate the requirement for
> "incremental deployability" are *the same* structural constraints that
> absolutely dictate that any/every conceivable current and future routing
> system that could ever be adopted and deployed in this manner will --
> inevitably, necessarily -- embody some set of features that ultimately
> make all such systems vulnerable to some version of *the same* systemic
> risks to which the current system is vulnerable, namely:

IOW: a good scalable routing solution can improve scalability within
IPv4, but as far as we know, can't help any one network or all uses,
bust out of IPv4 dependence to use some other system, such as IPv6.

To achieve such an outcome may well be impossible.

> -- "inflation" (c.f., generalized/cumulative/direct impact of individual
> demands on routing system carrying capacity that are borne by all/other
> participants in that routing system)

These financial system analogies strike me as rather loose.

End-user networks adopting SPI space place no burden on the BGP
control plane whatsoever.  There is a burden, from each MAB, but that
is not related to how many end-user networks share that MAB, how many
micronets they split the MAB into, or how frequently they change the
mapping for those micronets.

With Ivip, mapping changes are paid for by the end-user network - so
they are not an unreasonable burden on the RUAS.  (This is not a
perfect system, but it goes a long way to paying for the cost the
mapping changes impose on all parties which need to carry it.)
Indeed, these charges are a source of revenue and MAB companies would
compete in part by offering lower fees which still enabled them to
run a profitable business.

Hopefully all this will be clear if you read:

  http://www.firstpr.com.au/ip/ivip/Ivip-summary.pdf and

  Ivip business models: fast push & OITRDs  2008-04-18
  http://www.ietf.org/mail-archive/web/rrg/current/msg02100.html

> -- "deflation" (c.f., generalized/cumulative impact of non-availability
> of routing system capacity and/or address resources, which manifests as
> diminished growth potential for existing  participants and a progressive
> barrier to the emergence of potential new participants)

IOW: Shortage of address space.  This will still happen with Ivip or
whatever, but it will happen later due to the ability to slice and
dice the address space into small increments which can be used
efficiently due to being very useful to many end-user networks and
profitable for MAB companies when rented at a cost end-user networks
are happy to pay.

> -- "stagnation" (c.f., generalized/cumulative impact of non-availability
> of "standard"/"DFZ" routing system capacity and/or "standard"/uniform
> address resources, which manifests as a progressive degradation of and
> loss of confidence in the consistency/predictability of precisely those
> "network effects" that make a shared, inter-domain routing system so
> valuable to its direct participants, as well as to indirect stakeholders).

IOW: When existing and new companies find it harder than is ideal for
them to get address space, due to shortages, hoarding, unfair
distribution systems etc.

> artifact of current routing and addressing technology. 

I disagree.

The general problems of IPv4s address space being finite and less
than would be ideal are a totally separate matter from the problem of
unscalable growth in the burden of work on DFZ routers, together with
the inability of those so burdened (DFZ router operators) to recover
costs from those who impose the burden.  This growth is primarily due
to the fact that at present, without a scalable routing system, the
only way an end-user network can get multihoming, TE and portability
is to get its own PI prefix (one or more /24s - is a /22 a typical
allocation?) and then to advertise this as one or more prefixes (/24,
/23 etc.) in the global BGP system.

A core-edge separation scheme such as LISP, APT, Ivip or TRRP
(Six/One Router too, but just for IPv6) would change all that, since
there would be this new kind of SPI (my term, not used in LISP etc.)
space which would suit the needs of large and small end-user networks
much better than getting their own AS, PI prefixes etc.

Maybe there is some aloof framework from which you can see the
routing scaling problem as a specific instance of some more
generalised set of circumstances.  To me it is a specific set of
circumstances which occurs due to BGP's finite ability to cope with
large numbers of prefixes and frequent updates to those prefixes.  It
is the same in principle for IPv4 or IPv6, but there is no problem
yet for IPv6, and the IPv4 scaling problem happens to be occurring at
a time when fresh IPv4 space is running out.

It so happens that a routing scaling solution will, in my view,
enable much better utilisation of IPv4 address space.  How much time
that will buy us, I am not sure.  It depends a lot on population
growth - which can't be assumed given global warming's effects - on
Internet usage, and on whether cell-phone networks want to use a
global unicast IPv4 address for each device, or whether they will use
IPv4 NAT or IPv6 instead.

> Rather, these
> risks are a natural, organic, and (probably) largely unavoidable feature
> of *all* liquidity mechanisms -- i.e., all "technically mediated
> exchange systems." They are unavoidable because a necessary condition of
> possibility for the existence/emergence of such mediated exchange
> systems is the prior existence of a population of diverse, independent,
> often competing economic agents (individuals, institutions, etc.),
> members of which might often enjoy opportunities to productively
> interact with one another, but for the existence of various
> "transactional frictions" (foregoing details on this for now).
> distinguished by its capacity to reduce if not eliminate those frictions
> for all participants who voluntarily adopt it -- thereby becoming
> participating members in that particular liquidity system. The intrinsic
> friction-reducing benefits of participation (growth, profit, etc.)
> attracts and retains participants within that system, which
> creates/strengthens incentives for system participants to interact with
> each other ever more intensively (producing as a result the devision of
> labor, specialization, etc.). However, those same benefits also have the
> organic effect of intensifying competition between system participants,
> which eventually results in their becoming aware of the liquidity
> mechanism itself, and its possible uses as a strategic,
> (anti)competitive tool.
> 
> I'll spare the list the rest of the evolving story. Those interested can
> find more of it is on my personal website (links below).

I think there are some interesting parallels between IP addressing
and currencies, but I am not sure that they have so far provided me
with any important insights which help me solve scalable routing or
the larger problem of transitioning to a system without oppressive
address space limits.

> In any event, I wholeheartedly share RRG's ambition to contribute to the
> alleviation these problems. The story above explains why systems like
> ours need liquidity mechanisms, and what benefits they confer -- and
> risks they impose -- on those systems that are lucky enough to have one.

> However, that "need" does not entail that it will be fulfilled.

I understand this as the purported "need" to solve the IPv4
addressing limitation problem does not mean that this will be achieved.

I am sure humanity will cope with being boxed into IPv4 indefinitely.
 It is not ideal, but it is not necessarily disastrous either.  So I
don't regard it as a true "need" like our need for air, water, food,
space, the right temperatures, social and political freedom and
structures etc.

> Obviously, another absolute condition of possibility is the existence of
> some candidate mechanism(s) that is capable of initiating and sustaining
> this liquidity process over time. Not all candidate liquidity mechanisms
> are created equal, and I think of RRG's core mandate as the definition
> of the "best possible" successor liquidity mechanism for the Internet
> that we have inherited -- with "best possible" defined as (max.
> [probability of adoption * probability of sustainability, given known
> structural and exogenous risks]). 

I am not sure if RRG folks would recognise the group's goals if the
encountered them unawares expressed in those words!

I think the RRG's role is to provide a scalable routing solution
which achieves something like the three goals A, B and C in my footnote:

I also suggest we aim for D and E (Mobility and open-ended
flexibility), but these are not officially part of the RRG's goals.

> That said, I do not believe that the
> problems that RRG is trying to solve are amenable to any complete or
> final "solution" -- nor even to much symptomatic relief unless the root
> cause(s) of the problems are recognized and addressed directly.

Do you think there is something important we haven't recognised, and
that when we do, we will be able to solve the larger problem of
migrating from IPv4 to something more expansive?

> And that brings me back to the point of departure, Robin, which was the
> contrast between "desire" or (self-asserted) "need" vs. *eligibility*.
> It seems to me that the formula that made it possible for the last
> RRG-relevant domain-wide solution (CIDR) to "work" -- i.e., to be both
> (1) accepted by, at least, the minimum set of parties required to get
> started, and (2) to continue to be regarded as tolerable/acceptable by
> the majority of direct, indirect, and aspiring stakeholders thereafter
> -- involved three key ingredients:
> 
> justifiable/defensible set of criteria that can be used to determine an
> institution's "eligibility" to consume some share of the common/default
> routing system's finite carrying capacity. To maximize sustainability
> over time, these criteria should establish a link between
> individual-level demand for participation and the potential system-wide
> benefits of allocating a share of finite system carrying capacity to the
> aspiring participant.
> 
> capacity. 

I am wary of using the term "carrying capacity" to refer to the
resource of address space.  To me, carrying capacity is to do with
moving packets physically.  The question of address space quite a
separate set of limitations on the entire network.

> Achieving this in a sustainable way is only possible if the

You mean you don't believe that the Market always produces the best
outcomes for all????

This is not Capitalism's finest hour, and I don't have ultimate faith
in markets, but I don't see them as useless or inherently prone to
failure.

I think your previous paragraph asserts that acceptable outcomes can
only arise for all participants, in the case of IPv4 address space
allocation at least, if there is is some overarching, "fair" (however
defined) system which does not at all resemble a "market".

I think there is some truth in this.  For instance, if the allocation
system gives 90% of the space to 10% of the applicants, then the
outcome is bound to be unacceptable for most.

But let's consider the power of a market to achieve not so bad
outcomes, even from a lousy starting point.

Let's say Ivip or whatever enables MAB companies to obtain (beg,
borrow, cajole, purchase or rent) space and rent it out profitably as
SPI space.

MAB companies can get address space from anywhere, rent it to anyone
and need no physical infrastructure (they can pay other companies to
run OITRD and to handle the mapping).

So there is going to be a great competitive market developing between
MAB companies and all the end-user networks for whom the costs of SPI
space (fairly low) make it worth getting, for some combination of
multihoming, TE and/or portability between ISPs.

Consider yourself as a MAB company, and me as another one.  We are
renting out space at a rapid rate and our ability to do so profitably
depends in large part to our ability to get new space for this purpose.

Companies like yours and mine are able to turn IPv4 space into a cash
cow.

Now consider some ISP ISP-A or AS end-user network NET-B.  They have
currently got a bunch of space.  That space is worth something to
them, and they have certain costs.  If the way they use their space
is less profitable per IPv4 address than what your MAB company or
mine is ready to pay for it, then it is in their interests to move
their usage into a corner of their existing space and rent or sell or
whatever the rest of the space to a MAB company like yours or mine.

This new ability of the MAB companies to use space very effectively
means that market forces alone will cause companies to be motivated
to compact their usage to similar efficiency and then sell whatever
space they no longer need.  (This would be dependent on RIR policies
allowing sale or rental of space.  I am not assuming that would be
the case - but one way or another, if someone has a bunch of stuff
they don't really need and someone else needs it and is willing to
pay for it, that thing is going to go one way and money the other
before long.)

Your gloomy model seems to assume that all address space is owned by
ISPs, and that an ISP-X relinquishing space to an ISP-Y is
necessarily going to loose out.  But ISPs only compete in defined
geographic regions.   If an ISP in Melbourne Australia gives some
space to an ISP in Borneo, there is no change in competitive advantage.

The MAB companies do compete globally, so they will be fighting to
get space as long as they can rent it out profitably.

I think the combination of:

   Greatly increased ability to use IPv4 space close to 100%
   efficiency.

   A market for this in the form of non-mobile end-user networks
   wanting multihoming, TE and portability.

   A further potentially vast and lucrative market for mobile
   end-user networks (cellphones / PCs) wanting just a single
   IPv4 address they can use anywhere via the TTR Mobility
   system.

   A market for advertisable prefixes of global unicast IPv4
   space developing by hook or by crook.

will enable a gradual but over time thorough saturation of IPv4 usage
with generally stable cost per IPv4 address, due to the global demand
from MAB companies competing with ISPs and the large PI using
end-user networks for available space.

The more end-user networks use SPI space, the less space an ISP needs
to run its network as some proportion of its customers convert from
PA space to to SPI space.

So it is possible to imagine many end-user networks being on MAB
space, the rest being on PA space, and so the ISPs having all the
space not devoted to MABs.  That space would be largely devoted to
DSL etc. PI customers and the rest would be used for ETR and ITR
addresses.   This is simplistic, but it illustrates how market forces
would encourage networks which were not making much money from their
address space to rent or sell it to those who can.

There would still be iniquities - such as if those who happened to
get a huge amount of space were able to profit by renting or selling
it.

My point is that these market processes, while not ideal in principle
or in practice, would to quite a large extent provide sufficient
space to the ISPs and MAB companies who need it, because they would
be able to make money from this space - and so could afford to buy or
rent it from those who are using it less intensively and presumably
less profitably.

> So, to your point about not being able to "provide multihoming etc. for
> all end-user networks which want it," I respond that that is not a
> nature. 

I was using "want it" as shorthand for "want it sufficiently to pay
the price, which would be lower than the only current option: an AS
and a PI prefix".  I like this better than "need" it, because it is
easy for people outside the network to say that some end-user network
doesn't really "need" multihoming or whatever.

If Ivip or something likes it succeeds, it would have the effect that
SPI space and its benefits be generally available at a price that is
attractive to many more end-user networks than would currently be
able to afford PI space, even if it was available.

> I seriously doubt that it ever will be, because in an uncertain
> world full of real and/or potential competitors, people will always
> "want" more than they currently "need," and nothing short of infinite
> carrying capacity or a change in human nature can remedy that. 

Sure, I was trying to be concise by using just "want".

> However,
> I believe with even greater seriousness that we must aspire to preserve
> -- or barring that possibility to recreate with the shortest humanly
> possible interruption -- those three conditions, and the kind of
> "eligibility-based" system admission process that results when those
> conditions are in place.

I know little about the machinations of RIRs.  I imagine that already
and increasingly in the years to come, some highly sophisticated and
enlightened policies are being developed and enforced regarding who
gets to keep whatever space was previously allocated them in the
prior days of Plenty, when it was assumed that a much lower rate of
actual utilisation was all that was practical.  If Ivip or similar
works as I intend it then this will set a new high standard for close
to 100% utilisation of IP addresses.  I expect many end-user network
will be happy with one, a few or a relatively small number of
addresses whereas in the past, they might have been able to justify
getting a /22 or similar.

> (effectively no) time, whatever endures may not be redeemable, or worth
> redeeming for that matter, and in any event system maintenance by then
> is then likely to have become someone else's problem. 

This is pretty bleak.  I don't see any great disaster occurring in
IPv4.  I just think we will develop means of using the space more and
more intensively.  A scalable routing solution such as LISP, APT,
Ivip or TRRP would be a crucial part of this.

> Just as the "need"
> for a liquidity mechanism like TCP/IP does not guarantee that one will
> emerge and be widely adopted, neither does that need guarantee that an
> existing mechanism will endure, or in enduring superficially ("in form
> only") continue to provide those substantive benefits that make
> liquidity systems so valuable, and so vanishingly rare in the wild.

I agree that systems don't automatically run the way most people
would like them, but I don't foresee any great crunch in IPv4, unless
perhaps everyone insists on having a cellphone with its own dedicated
IPv4 address.  That is perfectly possible with the TTR Mobility
approach - but there probably won't be enough IPv4 addresses to go
around.  That won't be a sudden thing.  Humanity will cope with
cellphones not having their own IPv4 address!

> universally been backed by sovereign power -- and 100% of those were/are
> exclusively national in scope. What that portends for the future of our
> own, first-of-its kind, extra-territorial, attachment-based liquidity
> mechanism is left as a exercise for the reader.

Hmmm - it is intriguing to think that this uncommonly well designed
networking protocol actually created something like an international
currency.  IPv4 address space futures anyone?

> Thanks again for your responsiveness...

The Internet will be fine.  It has an endless supply of recycled
electrons.

The time, interest and attention span of RRG participants is another
matter - it is a precious resource.  There should be a refreshment
stall here for people who made the distance.

That unpleasant prickling sensation you feel is one or two RRG folks
sticking pins in your effigy every time you write more than about 4
paragraphs in a list message.  Maybe they will give mine a bit of a rest.

  - Robin

---

From: Tom Vest <[email-masked]>
Date: Mon, 20 Apr 2009 07:40:42 -0400

Hi Robin,

Thanks again for the very detailed response; it is flattering whenever  
anyone take the time to do something like this, even if the response  
is substantially or entirely critical. So, thank you once more.

I can certainly understand how one might arrive at an initial  
impression that the "liquidity system" view provides no new insights  
for how to structure RRG's efforts, and even how a cursory or partial  
reading might leave one with doubts about its overall validity. I will  
do my best to challenge those impressions as soon as I get past a  
couple of urgent work assignments, hopefully in the next 24-36 hours.

In the mean time, I will simply remark that the theory did not emerge  
with RRG in mind, but rather has been developing over the last six  
years, initially based on my hands-on experience as a very large  
operator working across many different material, commercial, and  
regulatory environments, at a level of the industry where largely  
unconstrained strategic bargaining plays a substantial role in shaping  
some relatively inflexible decision making constraints for the rest of  
the industry (and before that as a very well-placed but passive  
observer of the topmost levels of the international telecom and  
monetary policy-making spheres, and after that as an advisor to  
protocol resource coordinating institutions, regulators, and private  
investors). The theory is not narrowly conceived in instrumental  
terms, along the lines of the RRG charter, but rather is an attempt to  
integrate a very broad range of empirical regularities in a  
meaningful, illuminating way. Assuming that they continue to hold up  
to ongoing scrutiny, recognition  of and theoretical engagement with  
those regularities *should* usefully inform the kind of work that of  
RRG is chartered to undertake, if only by illuminating any  
hypothetically interesting development trajectories that have proven  
to be consistently productive and/or unproductive under "similar"  
conditions in the past. I believe that the theory does satisfy, and  
will continue to satisfy that standard -- and perhaps do even more --  
but I recognize that the burden will be mine, perhaps indefinitely, to  
continuously illustrate and defend those claims.

Of course, even if the theory manages to live up to that challenge  
forever, there is no guarantee that it will deliver a specific  
solution that can meet all of the looming challenges without  
sacrificing any of the features that have made the Internet so  
distinctive (and I would add, important) to date. I would like to  
think that it could, and might yet, but failing to do that would not  
invalidate the theory, or even its potential relevance to RRG. To  
quote a favorite movie, we are in a "tight spot"... I'd very much like  
to see us get out of that spot without making any irreversible  
tradeoffs that we might sorely (but impotently) regret going forward.

So, look for a more substantive response in the next day or two... and  
thanks again.

Regards,

TV