Subject: Re: [RRG] Moving forward... IPv4 now, IPv6 less urgent and perhaps more ambitious

From: Robin Whittle <[email-masked]>
Date: Tue, 10 Jun 2008 05:36:21 +1000

I understand the current draft text is:

> Our recommendation should be applicable to IPv6.  It may or may not also
> apply to IPv4, but at the very least must provide a path forward for IPv6.

I oppose this because it would allow the RRG not to recommend a
solution for IPv4.

I agree with a lot of what Scott, Dino and Christian wrote.

There is currently only one routing scaling problem - in the IPv4
Internet.  There's no hurry about solving the IPv6 routing scaling
problem because it will only occur if and when IPv6 is widely adopted.

AFAIK we are not required to assume that IPv6 adoption (without IPv4
addresses, meaning mass migration from IPv4) will occur soon enough
that the IPv4 routing scaling problem does not need to be solved.
RAWS concentrated on the IPv4 scaling problem.

The world in general uses IPv4.  Enthusiasm for IPv6 is limited and
is much more prevalent in the IETF than amongst most users or
providers.  (IOW, since its development in the mid-1990s, IPv6 has
not come close to meeting the needs of significant numbers of users
or ISPs.  There is no evidence that the looming IPv4 address
shortage has changed this.)  We should consider that such confidence
about IPv6 is likely to involve unrealistic expectations of imminent
dual-stack and IPv6-only adoption.

If the RRG only recommends an IPv6 solution, almost no-one will take
us seriously or care what our recommendation is.  Almost no-one will
devote serious effort to solving a problem which does not yet exist
and will not arise for 10 years or so.  This is especially so while
the IPv4 problem needs to be solved ASAP and will get significantly
worse as space is sliced and diced more finely after 2010.

I would support text such as:

  We should recommend a solution to the IPv4 routing scaling problem
  which has the best possible chance of being widely adopted in the
  time-frame 2012 to 2015.

  In order for the solution to be widely enough adopted, it must
  involve minimal costs and disruption and provide significant
  immediate and lasting benefits to the new and existing end-users
  and providers who adopt it.  This rules out changes to host stacks
  or applications except perhaps to cope better with any small
  performance limitations imposed by the solution itself.

  We should also recommend a solution to the IPv6 routing scaling
  problem, although with a potentially longer time-frame for
  development and deployment.  Our recommendation for IPv6 may
  be more tentative or suggest multiple avenues which should be
  researched further before a final decision is made.

  The IPv6 situation differs in several important respects
  from the IPv4 situation, including:

    1 - Greater possibility of host stack and application changes
        due to lower installed base.  So perhaps the solution
        could involve fundamental changes and/or additions to the
        IPv6 protocols.

    2 - Likewise, greater possibility of changes to routers, the
        DNS system and other items of network infrastructure etc.
        (Maybe BGP for IPv6, but I think we all agree BGP can't
        improved enough to solve the routing scaling problem on
        its own.)

    3 - The longer address being used potentially for very different
        address management policies and in specific ways for the
        new protocols.  (eg. SHIM6, Six One, Six One Router.)

    4 - Less urgency about development and deployment.

    5 - Therefore, the possibility of choosing and developing
        the IPv6 solution after the IPv4 solution has been
        developed and perhaps partially deployed.

    6 - Therefore the possibility that the solution may also
        be desired or required to explicitly support additional
        goals, such as mobility, QoS across the global Internet,
        better security etc.  (This includes the improvement of
        IPv6 over IPv4 for the explicit purpose of encouraging
        faster IPv6 adoption and so facilitating a more
        rapid changeover by most users from IPv4 and dual stack
        to IPv6 only.)

  Consequently, our recommended IPv6 solution may differ
  significantly in detail and perhaps in principle from the IPv4
  solution.

  However, any difference in principle between the two solutions -
  especially in terms of the types of new network functions (ITRs,
  ETRs etc.) and where they are located - should be minimised
  and justified fully by the advantages which such differences
  enable.

  Likewise, any extensions to the IPv6 protocols which would affect
  applications (such as changes which make it harder to convert IPv4
  applications to IPv6) would need to be fully justified by
  important benefits.

I think that map-encap is the best solution for both IPv4 and IPv6.
 Ivip for IPv4 and IPv6 should be identical in principle and in most
details.

However, I can't rule out that there may be some fancier, better and
more ambitious approaches to solving the IPv6 routing scaling
problem.  This includes fundamentally different approaches,
especially if we contemplate changing the IPv6 protocols.

 - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "Tony Li" <[email-masked]>
Date: Mon, 9 Jun 2008 13:51:45 -0700

Robin,

|There is currently only one routing scaling problem - in the IPv4
|Internet.  There's no hurry about solving the IPv6 routing scaling
|problem because it will only occur if and when IPv6 is widely adopted.

I have a lot of sympathy for where you're coming from, but I have to
disagree.  Yes, we have one routing scaling problem, but it's the IP routing
architecture, and it's common for both v4 and v6 because both use exactly
the same routing architecture.  There's no hurry to fixing it for IPv4
because frankly, we can scale the hardware to support the usual PI prefix
sizes (/24 -> 2^24 routes == 16M routes).  Yes, it will cost a bit more, but
over the long run, that's probably a lot cheaper than upgrading every CPE or
PE router.

Where we have major issues is (and, admittedly, if we transition to) v6.
Since the canonical PI prefix is a /48, we will have a very serious problem
on our hands.  Now, I'm no lover of v6 and I think my views on it are pretty
well known.  While you may not take v6 very seriously, there are those that
do, and frankly it's our joint responsibility to plan for its adoption.  We
have had one 'success disaster' with v4 and we need to prevent the same
thing from happening to our grandchildren.  I'll happily stipulate that we
don't know when a solution will be necessary.  V6 deployment has been about
15 years running so far and shows no signs of reaching critical mass anytime
soon.  However, this is also partly why this is considered a _research_
topic, not an engineering one.  

|However, I can't rule out that there may be some fancier, better and
|more ambitious approaches to solving the IPv6 routing scaling
|problem.  This includes fundamentally different approaches,
|especially if we contemplate changing the IPv6 protocols.

... And that's exactly what we are chartered to explore.

Tony

---

From: David Conrad <[email-masked]>
Date: Mon, 9 Jun 2008 14:57:06 -0700

On Jun 9, 2008, at 12:36 PM, Robin Whittle wrote:

> ...
> ...

Um. 433 words instead of 28?  In order to reach rough consensus on the  
direction the group is taking?  Do you, perhaps, get paid by word?  :-)

More seriously, while I understand some of your concerns (and disagree  
with others), I think we need to plant a stake in the ground to  
demonstrate at least some progress has been made.  The statement Tony  
provided is pretty watered down with respect to saying much of  
anything, but it at least points to a prioritization and one that I'm  
comfortable with.  As such, I'm OK with the wording as is (although I  
might prefer "Our recommendation must be ..." since it "must" appears  
in the second sentence).

Regards,
-drc

---

From: "William Herrin" <[email-masked]>
Date: Mon, 9 Jun 2008 23:07:09 -0400

> I oppose this because it would allow the RRG not to recommend a

For what it's worth, I agree with Robin. The identified problem is:
find a way to drastically reduce the $8000/year cost of each IPv4
prefix in the core. Shifting focus to IPv6 abandons the problem.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked] [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Xu Xiaohu <[email-masked]>
Date: Tue, 10 Jun 2008 12:10:13 +0800

> ... And that's exactly what we are chartered to explore.
 
Hi Tony,

I agree with your proposed statement. BTW, should we add "solving the
address depletion issue" into the RRG design goals document as a desired
goal and not mention IPv6 specifically and directly?

Best regards,
Xiaohu XU

---

From: Robin Whittle <[email-masked]>
Date: Tue, 10 Jun 2008 14:42:28 +1000

I am replying to Tony Li and David Conrad - and requesting the views
of the LISP and APT developers on my proposed text:

  http://psg.com/lists/rrg/2008/msg01444.html

Tony wrote:

> the same routing architecture.  

There is a potential routing scaling problem because the IPv4 and
IPv6 routing systems are identical in respect of the limitations
which give rise to the problem.  However the IPv6 system does not
have the physical problem so far because very few people use it.

This is a routing and addressing scaling problem and I think it is
is only solvable by creating some new kind of address space which
provides multihoming etc. in a scalable way.

The fact that IPv4 and IPv6 use exactly the same routing structure
does not mean that the solution needs to be the same, because there
are significant differences in the addressing systems:  IPv6 has 96
more bits and a vast, currently largely uninhabited space.

This enables that compared to an IPv4 solution, an IPv6 solution can
involve different assignment policies and new protocols making more
creative use of address bits.  For instance IPv6's larger address
space makes it practical to have duplicate address ranges going to a
translation router, such as with Six One Router: one from each
upstream ISP, each of the same length as the end-user prefix.  See:
http://psg.com/lists/rrg/2008/msg01314.html

> There's no hurry to fixing it for IPv4

If your view reflects consensus in the RRG, then perhaps we could
form rough consensus on something like this:

  While some architectural enhancements could solve the routing and
  addressing problem by creating a new kind of address space which
  is portable, multihomable etc. without significantly burdening the
  BGP system, we believes that none of them could be practical or
  cost-effective enough to warrant development or deployment for
  IPv4, compared with letting the current system saturate towards
  16M individually routed /24s.

If so - and if I believed this was a viable solution - then I would
mothball Ivip until about 2020.

You are suggesting the current DFZ size of 250k:

  http://bgp.potaroo.net   Ooops make that ~260k . . .

should be allowed to grow a factor of up to 64 without any changes
in the BGP system or any architectural enhancements.

If this could be done, then we could split up the IPv4 space finely
and make pretty good use of it, without any new map-encap scheme
etc.  Quite a large number of end user networks could work fine with
a /24.

However, map-encap would be much better because a much larger number
of end-user networks could scalably have their own portable,
multihomable space, in any address size - 1, 2, 4 etc. through 256
IP addresses and beyond.  The right map-encap system could also
support a new global form of mobility.

I understand that for considerable, but perhaps not unthinkable,
expense the 200k+ (and more in the future) DFZ routers could be
upgraded to have FIBs to handle this.  My first suggestion (early
2007, now abandoned) for the routing scaling problem:

  http://www.firstpr.com.au/ip/sram-ip-forwarding/

was just this, when I thought that the FIB was the problem, not the
RIB and the associated BGP communications.

My understanding of your suggestion is that it is much the same as
my now-abandoned solution: flat routing to /24 with souped up
SRAM-based FIBs and hoping that Moore's Law would affordably provide
the extra CPU and RIB RAM grunt required to handle the up-to 64
times increase in the number of conversations each DFZ router needs
to have with each of its peers.

No-one would have gone to the trouble of developing a map-encap
proposals if we thought that it was acceptable to let the IPv4
system degenerate as you suggest without any new architectural
system for taking the load off the BGP system.

None of the map-encap systems were made with the intention that IPv4
be left to fester in a swamp ten or more times bigger than its
current predicament.  They are all described initially in terms of
fixing IPv4.

Surely enough was agreed at RAWS about problems with BGP stability
and the load on CPUs and RAM to make your suggestion of leaving IPv4
alone a non-starter.

> do, and frankly it's our joint responsibility to plan for its adoption.  

I totally agree we should plan for widespread IPv6 adoption.  My
text reflects the facts that the IPv6 solution is not so urgently
required and that an IPv6 solution could take advantage of a number
of IPv6-specific conditions which unfortunately don't apply to IPv4.

> We have had one 'success disaster' with v4 and we need to prevent the same
> don't know when a solution will be necessary.  

OK - we don't yet know when an IPv6 solution will be necessary.

> V6 deployment has been about

As I suggested, I think we would be doing our job well if we
suggested several promising approaches to IPv6, since there is no
urgency about deciding on which should be developed and deployed.  I
think the situation with IPv4 is urgent enough that we should find
the most promising solution, or class of solutions, and recommend
that for immediate development.  Perhaps we won't reach consensus on
one, but there may be considerable support for two solutions.  If
so, that is the best we can recommend.  If we can't reach much
consensus on anything, then I think our report should reflect that.

> ... And that's exactly what we are chartered to explore.

Yes, but I would be surprised if there is consensus that the IPv4
routing and addressing problem is so inconsequential that we don't
need to recommend a more specific and better researched solution for
IPv4 than we do for IPv6.

David Conrad wrote:

> Um. 433 words instead of 28?  In order to reach rough consensus on
> the direction the group is taking?  Do you, perhaps, get paid by
> word?  :-)

I wish.

> More seriously, while I understand some of your concerns (and
> disagree with others), I think we need to plant a stake in the
> ground to demonstrate at least some progress has been made.

Yes, but not if the statement involves shortcuts that fence us off
from the optimal combination of recommendations.  Nor if the
statement is insufficiently meaningful to constitute progress, for
instance by not eliminating any significant part of the solution space.

Despite its length, I think my text has a good signal-to-noise ratio
because it:

  Usefully fences off certain parts of the solution space for IPv4
  (host changes, except perhaps for minor performance enhancements).

  Emphasises that the IPv4 solution needs to involve immediate and
  lasting benefits over costs for ISPs and end-user networks.
  (Implicitly, this is because without coercive powers, the only
  way the IETF can ensure sufficiently widespread adoption to
  substantially solve the problem is by making the solution directly
  attractive according to costs and needs of a large majority of
  potential adopters.  Almost no-one will adopt it simply because it
  is a solution to the scaling problem if it involves any costs or
  disruption, which it certainly will.)

  Commits us to devising an IPv4 solution for immediate development
  and for deployment in what I think is the earliest practical
  time frame.

  Gives us more flexibility in the IPv6 solution space, because it
  is a broader space and because there is no urgency.

> The statement Tony provided is pretty watered down with respect to
> saying much of anything, but it at least points to a
> prioritization and one that I'm comfortable with.  As such, I'm OK
> with the wording as is (although I might prefer "Our
> recommendation must be ..." since it "must" appears in the second
> sentence).

OK.  Based on recent messages of support for Tony's text, I am
surprised by how many people seem to think we don't need to solve
the IPv4 problem.

Where are the LISP and APT developers on this question?  (Bill
Herrin supports my text.)  I think we all went to a lot of trouble
to develop our solutions because we believe something really is
needed ASAP for IPv4.  That was the impetus behind the RAWS workshop
20 months ago.

In my view, there are no other potentially practical solutions for
IPv4 than the map-encap systems.

If the map-encap developers are outvoted in some sense by other folk
who believe there isn't a need to solve the IPv4 routing scaling
problem, then I think that the consensus view would have resulted
from the views of people who haven't yet devised an IPv4 solution
prevailing over those of people who think they have.

If so, folks outside the RRG - who are seriously concerned about the
IPv4 problem in terms of costs and stability now and in the years to
come - will regard the IPv6-centric work of the RRG as irrelevant.
If this eventuates, then the RAM list and later the RRG list will
have been productive in terms of supporting the development of a
bunch of map-encap solutions for IPv4 which the RRG chose not to
recommend.

 - Robin

---

From: "Tony Li" <[email-masked]>
Date: Mon, 9 Jun 2008 22:01:14 -0700

Hi Xiaohu,

|I agree with your proposed statement. BTW, should we add "solving the
|address depletion issue" into the RRG design goals document as 
|a desired
|goal and not mention IPv6 specifically and directly?

Could you concisely state your understanding of the issue?  To my
understanding, the only address depletion issue is the usage of the v4
address space, and somehow, I think you're thinking of something else.

Tony

---

From: "Tony Li" <[email-masked]>
Date: Mon, 9 Jun 2008 22:36:21 -0700

Hi Robin,

|There is a potential routing scaling problem because the IPv4 and
|IPv6 routing systems are identical in respect of the limitations
|which give rise to the problem.  However the IPv6 system does not
|have the physical problem so far because very few people use it.

So you suggest that we wait until the problem is critical before we fix it?

|If your view reflects consensus in the RRG, then perhaps we could
|form rough consensus on something like this:
|
|  While some architectural enhancements could solve the routing and
|  addressing problem by creating a new kind of address space which
|  is portable, multihomable etc. without significantly burdening the
|  BGP system, we believes that none of them could be practical or
|  cost-effective enough to warrant development or deployment for
|  IPv4, compared with letting the current system saturate towards
|  16M individually routed /24s.

That seems a bit overboard.  Precluding possible good ideas doesn't seem
like it makes sense either and we certainly can't be sufficiently assured of
our position to say that "none" of them could be practical.  Doesn't that
seem like it's a bit absolutist given that we've barely scratched the
solution space?

|You are suggesting the current DFZ size of 250k:
|
|  http://bgp.potaroo.net   Ooops make that ~260k . . .
|
|should be allowed to grow a factor of up to 64 without any changes
|in the BGP system or any architectural enhancements.

I'll just point out that the table has grown by a factor of 50 since I
started working in core routing.  ;-)

|I understand that for considerable, but perhaps not unthinkable,
|expense the 200k+ (and more in the future) DFZ routers could be
|upgraded to have FIBs to handle this.  My first suggestion (early
|2007, now abandoned) for the routing scaling problem:
|
|  http://www.firstpr.com.au/ip/sram-ip-forwarding/
|
|was just this, when I thought that the FIB was the problem, not the
|RIB and the associated BGP communications.

As I think we've noted elsewhere as well, the RIB issues will continue to
scale upwards with ever increasing costs and complexity at the BGP level as
well.  More generally, it seems like most scaling problems can be
forestalled by the extreme application of cash.  This suffices as a short
term strategy while we produce a clean longer term solution.

|My understanding of your suggestion is that it is much the same as
|my now-abandoned solution: flat routing to /24 with souped up
|SRAM-based FIBs and hoping that Moore's Law would affordably provide
|the extra CPU and RIB RAM grunt required to handle the up-to 64
|times increase in the number of conversations each DFZ router needs
|to have with each of its peers.

Not necessarily completely flat to /24.  Rather, we could just allow v4 to
play out organically and let the current system of checks-and-balances cause
some signficant pain for others while we deal with the longer term issue.

|Surely enough was agreed at RAWS about problems with BGP stability
|and the load on CPUs and RAM to make your suggestion of leaving IPv4
|alone a non-starter.

As one of the primary protagonists at RAWS, I can assure you that there was
nothing said at RAWS that excluded that option.  In fact, I believe that
that exact option was raised at the time.

|OK - we don't yet know when an IPv6 solution will be necessary.

The eventual depletion of the address space suggests that it will happen,
eventually.  It may require many rounds of NATing and other such
contrivances before v6 seems less painful, but there is no obvious assurance
that we can make v4 viable indefinitely.  Thus, at some point, v4 will end
up being less attractive than v6 and some dual stack deployment would begin.

Note that fixing the routing architecture in v6 would aid its adoption.

|As I suggested, I think we would be doing our job well if we
|suggested several promising approaches to IPv6, since there is no
|urgency about deciding on which should be developed and deployed.  I
|think the situation with IPv4 is urgent enough that we should find
|the most promising solution, or class of solutions, and recommend
|that for immediate development.  Perhaps we won't reach consensus on
|one, but there may be considerable support for two solutions.  If
|so, that is the best we can recommend.  If we can't reach much
|consensus on anything, then I think our report should reflect that.

If we can't reach consensus, there's not much point in writing the report.
;-)

|Yes, but I would be surprised if there is consensus that the IPv4
|routing and addressing problem is so inconsequential that we don't
|need to recommend a more specific and better researched solution for
|IPv4 than we do for IPv6.

Exactly why I'm testing the consensus.

|OK.  Based on recent messages of support for Tony's text, I am
|surprised by how many people seem to think we don't need to solve
|the IPv4 problem.

And some of us are equally surprised that there is such a fervent desire to
provide a v4 solution.

The discussion of our basic underlying differences is the only way that we
will ever create movement within the community and thus reach consensus.  If
all we do is sit in our entrenched positions and endlessly reiterate the
benefits of the respective proposals, we are guaranteed to accomplish
nothing in establishing consensus.

|I think we all went to a lot of trouble
|to develop our solutions because we believe something really is
|needed ASAP for IPv4.  That was the impetus behind the RAWS workshop
|20 months ago.

I beg to differ.  The impetus behind RAWS was to fix the architecture, not
necessarily the protocol.  My personal agenda (shared by a few others, I
think) was to reopen the case for GSE, for example.  Please stop projecting
your views onto others that you don't represent.

Tony

---

From: Dan Jen <[email-masked]>
Date: Mon, 09 Jun 2008 23:38:20 -0700

Hi everyone,

On Tue, 2008-06-10 at 14:42 +1000, Robin Whittle wrote:
> Where are the LISP and APT developers on this question?

Here are thoughts from the APT team on this issue.

Though we feel that the RRG should recommend a solution which is 
practical and deployable today, certainly it must also be a suitably 
long-term, generic solution -- even as we move beyond IPv6 to IPv7, 8, 
and so on. With that in mind, we recommend the following:

"Our recommended solution should be address-format agnostic. That is, it
should be able to work with any reasonable addressing scheme. IPv4 and
IPv6 are examples of such addressing schemes."

-Dan and Michael

---

From: Xu Xiaohu <[email-masked]>
Date: Tue, 10 Jun 2008 14:39:13 +0800

Hi Tony,

I wonder whether the RRG is to design a new architecture or just to fix the
scalability issue of IPv6. And I think it will determine how much change we
can make in the Internet architecture.
In other word, should HIP like approach be taken into RRG's consideration
provided that there was no separate working group and research group in
IETF/IRTF? 

Best regards,
Xiaohu XU

---

From: Robin Whittle <[email-masked]>
Date: Tue, 10 Jun 2008 17:34:50 +1000

Hi Tony,

> So you suggest that we wait until the problem is critical before we fix it?

No - I suggest we don't rush into fixing a future IPv6 problem while
 the IPv4 problem costs hundred of millions (probably billions) of
dollars, restricts the availability of multihomable address space
and so needs fixing in the 2012 to 2015 time frame.

Since an IPv6 solution can be developed with a broader range of
techniques in a longer time frame, I suggest that by the time an
IPv6 solution needs to be decided upon, we should have a lot more
experience in the IPv4 solution than we do now.

This way, the IPv6 solution would not be frozen before it needs to
be and could benefit from IPv4 experience.  I guess that decisions
about how different the IPv6 solution is from the IPv4 one, and how
many other goals such as solution should support, need only be made
when the IPv6 DFZ routing table looks like it is going to become
problematic four or five years from that time.

> seem like it's a bit absolutist given that we've barely scratched the

My indented text above was my attempt to formalise what I understood
by what you wrote:

> There's no hurry to fixing it for IPv4 because frankly, we can
> scale the hardware to support the usual PI prefix sizes (/24 ->
> 2^24 routes == 16M routes).  Yes, it will cost a bit more, but
> over the long run, that's probably a lot cheaper than upgrading
> every CPE or PE router.

which I interpreted as meaning that there was no need to fix the
IPv4 problem at all.  I figured you meant to let the IPv4 system run
as is, to the point where there was finely divided address space for
PI space for many more end-user networks than at present, with
consequent scaling problems in the BGP system regarded as being
acceptable, due to them being "a lot cheaper than upgrading every
CPE or PE router."  I thought that by this you meant, in effect:

  a lot cheaper than upgrading every PE router, which is typically
  what happens with a map-encap scheme to build ETR and perhaps
  ITR functions into the ISP networks there.

So I understood your statement as meaning that the IPv4 problem was
not amenable to an acceptable solution by any means we currently
know of, including the map-encap schemes.  My understanding of this
position is that you are much more optimistic about IPv6 adoption
than I am.

> started working in core routing.  ;-)

Sure, but I think many people expect us to recommend a solution for
IPv4 in a time frame shorter than that those 15 or more years!

The size is approximately doubling every 4 years, but I expect this
time to shorten to maybe 2 years after about 2010 when IPv4 space is
much more finely and carefully divided than at present.

This cranks up the cost of DFZ routers - costs which flow to all
Internet users and place further pressures on how hard it is for an
end-user network to get PI space and advertise it as it needs to for
multihoming and TE.  This further increases the cost of running
end-user networks, which winds up burdening many end-users directly,
and the taxpayers and customers who indirectly pay the costs of
those end-user networks.

> term strategy while we produce a clean longer term solution.

Perhaps you could be more specific about years and outcomes.

My guess is you mean something like this, assuming the DFZ doubling
time reduces to 3 years after 2010, which I think is a very
conservative estimate:

        IPv4:                  IPv6:
        Do nothing and let
        DFZ size grow to:

  2009    310k
  2010    370k                 Develop solution
  2011    466k
  2012    587k                 Deploy solution
  2013    740k
  2014    932k
  2015   1175k                 But it is still cheaper and
  2016   1480k                 better to upgrade IPv4 DFZ routers
  2017   1864k                 than to try to run an ISP with
  2018   2350k                 customers only having IPv6
  2019   2960k                 addresses - so IPv4 remains
  2020   3730k                 the real Internet and IPv6
  2021   4699k                 adoption remains limited.
  2022   5920k
  2023   7460k

I think most (every?) ISPs and end-user network operator expects
there to be a solution to the IPv4 routing scaling problem and the
closely associated difficulty getting new address space in the 2010
to 2015 time frame.  (The current BGP-based address space management
system prevents full utilization by preventing fine splitting of
space into chunks smaller than 256 IP addresses or numerous enough
to meet the needs of hundreds of thousands, millions or billions of
end-user networks.)

With or without such a solution, I think it will still be impossible
to attract end-users to IPv6-only services, since such a service
can't provide all their needs.  Even if an IPv6-only service
provided for 90% of their needs, the support costs would make it
prohibitive compared to using NAT and finer slicing of IPv4

> some signficant pain for others while we deal with the longer term issue.

This is a good example of what I think is the extremely unrealistic
expectations amongst some folks about:

  1 - How easy it will be to get anyone to adopt an IPv6 service
      and run dual-stack with at least some major apps using IPv6.

  2 - How soon the proportion of hosts with IPv6 connectivity could
      get to the very high level (99%? 99.99??) required for
      ordinary end-users to be happy with an IPv6-only service.
      (This would presumably require most applications to have been
      upgraded to IPv6.)

  3 - The difficulty with which IPv4 space can be utilized much more
      efficiently than at present.  It is not that difficult with
      the existing NAT and BGP techniques, just ugly and expensive
      - but not prohibitively so compared to trying to sell an IPv6
      only service.  Map-encap and NAT would be less ugly and highly
      scalable.

I think this high level of optimism about IPv6's usefulness to
ordinary users, and about how rapidly it will be adopted, is held
primarily by people with close association with the IETF rather than
amongst ISPs or ordinary end-users.

That optimism has been around since about 1995 but has not yet been
reflected in end-users or ISPs wanting, needing or adopting IPv6.  I
don't think IPv4 routing or address difficulties are severe enough
to drive people to adopt IPv6 any time soon.

However convoluted and difficult English gets, it doesn't motivate
significant numbers of people to abandon the language and use
Esperanto instead.

I don't think the RRG can reasonably assume such optimism about IPv6
is realistic.  That is why I wrote:

  We can't bet the Internet on the imminent adoption of IPv6
  http://psg.com/lists/rrg/2008/msg01373.html

just under 2 weeks ago.

> that exact option was raised at the time.

OK - I wasn't there.

> up being less attractive than v6 and some dual stack deployment would begin.

I think you are advocating letting the entire Internet-using world
suffer and stew in an uncared-for IPv4 Internet until the costs are
so great that some end-users find it better to adopt a new Internet
which doesn't connect properly to the old one.

I think this is unrealistic and that the RRG can and should do better.

Here are two statements which I think are true now and will remain
true, despite the IPv4 address depletion problem, because there are
always going to be ways of incrementally splitting up IPv4 space
finer and using more NAT, which will be better than trying to sell
an IPv6-only service which doesn't do what people want, in
competition with other ISPs who sell a fully functional IPv4 service:

A - There's no reason for ordinary end-users to adopt IPv6 -
    dual stack or IPv6-only because it does not enable them to
    communicate with any host of interest which they can't already
    reach perfectly well via IPv4.  (China and cell-phone systems
    are significant possible exceptions.)  There will be no reason
    for ISPs to sell IPv6 only or dual stack services for these
    reasons - or for most end-users to adopt them if they were
    on offer - as long as most people's computers have an IPv4
    connection.

B   Most (all) end-users will use IPv4 until most (almost all?)
    end-users have a pure IPv6 or a dual stack service.

There's no way out of this as far as I can see.

If you propose that the RRG solution depend on mass adoption of
IPv6, I think you need to counter arguments such as mine about why
this won't happen in the foreseeable future, and propose some time
frames and mechanisms by which you think we can all be reasonably
certain the mass adoption and migration will take place.

> Note that fixing the routing architecture in v6 would aid its adoption.

I don't see why, since end-users don't care and ISPs won't care
until the IPv6 DFZ routing table became large enough that their
routers cost too much.  But even that is assuming a level of IPv6
adoption which I cannot imagine happening in the foreseeable future.

In the absence of widespread adoption of map-encap for IPv4 the
bloated and still growing IPv4 DFZ routing table would be driving
the rapid development of DFZ routers with huge RAM and CPU
capabilities.  Since these same routers would generally also be
handling IPv6 traffic on the same long-distance links, I find it
hard to imagine IPv6 adoption being so advanced as to lead to such a
large DFZ routing table size that the cost problems of fielding DFZ
routers for IPv6 traffic beyond the costs already involved in
fielding IPv4 DFZ routers would be a motivating factor for ISPs.

The above scenario applies if the IPv6 DFZ routing table grows to
rival the size of the IPv4 one - yet your argument is about a
routing scaling solution for IPv6 encouraging early adoption of
IPv6, well before the IPv6 DFZ routing table could have grown to
such a size.

> If we can't reach consensus, there's not much point in writing the report.

Except to outline our attempts and try to present the viewpoints of
the various sub-groups of people, none of whom were large enough and
none of which had sufficient commonality of views to form a consensus.

That would not necessarily a failing of the co-chairs or of the RRG
in general.  It may turn out that in 2008 to early 2009, amongst any
reasonably competent group of people in this field, that there is
currently not enough agreement on the problems, the urgency of
solution, the linkage to other goals or the merits of the various
suggested solutions to form consensus on anything - much less to
recommend any one solution, or architectural approach, for
engineering development.

> Exactly why I'm testing the consensus.

Yes - and why I am proposing an alternative which I think would be
agreed to by the map-encap developers at least, and hopefully many
other folks.

> And some of us are equally surprised that there is such a fervent desire to

And some of us are surprised that anyone is surprised that some of
us think it is important to keep the current Internet working, given
that:

 1 - There do seem to be ways of doing it scalably (map-encap).

 2 - Even without map-encap, it will still be kept going despite
     problems with routing scaling and with shortage of PI space
     for large numbers of end-user networks who would have been
     able to get it scalably with map-encap.

 3 - The only currently available alternative (IPv6) is not directly
     connected to or fully compatible with the IPv4 Internet, in
     terms of communications between hosts, in terms of application
     protocols or in terms of many existing applications.

 4 - While IPv6 (as in the past 13 years since its inception)
     provides nothing that end-users or ISPs actually want or need.

> nothing in establishing consensus.

I think the discussion has been productive because it has helped
both camps realise there is another camp with very different ideas -
whereas before, I think we tended to assume we were all pretty much
in agreement.

My proposed text is supported by Bill Herrin and somewhat supported
(IPv4 is a must, but with the same techniques as IPv6) by the APT team:

  http://psg.com/lists/rrg/2008/msg01458.html

Your position:

  Ok, take 2:

  Our recommendation should be applicable to IPv6.  It may or may
  not also apply to IPv4, but at the very least must provide a path
  forward for IPv6.

is supported by quite a few people, including Noel Chiappa, Joel
Halpern, Olivier Bonaventure, David Williamson, Ran Atkinson, Brian
Carpenter, Bob Hinden (what does WFM mean?) and Marshall Eubanks.

Fred Templin disagreed with your text, wanting IPv4 and IPv6
solutions.   Scott and I think Dino disagreed with an earlier version.

Unless there is a significant majority in favour of one or the other
positions - that we need to recommend an IPv4 solution or not - then
I think the RRG can't reach consensus on anything much, since the
answer to this question is a fundamental basis on which we make most
other decisions.  It affects the goals, the techniques available and
the time frame.

Fixing the IPv6 routing scaling problem in sufficient time is easy.

It may be as easy as doing absolutely nothing because maybe IPv6
will never be used enough to have a routing scaling problem.

Even assuming IPv6 is somehow widely used by 2020, it is an easy
problem to solve compared to IPv4, since there is no urgency, and
since there are lots of address bits to play with - and lots of
space to splash around.

Fixing the IPv4 problem is what the map-encap folks tried to do.
Each of us think we have achieved a promising approach, though we
favour our approach over the others.

If the RRG can't form consensus that any one such map-encap approach
 is suitable for IPv4, then so be it.

However, if that decision is based on what I regard as a completely
unrealistic notion of IPv6 adoption - as I think your position is
based on - then I think that consensus would be on a faulty basis
and would not be widely understood or respected by the rest of the
world who care a lot about IPv4 and have little or no reason to be
enthusiastic or optimistic about IPv6.

> your views onto others that you don't represent.

OK.

It is my impression this was the impetus behind RAWS, but I wasn't
there.  Perhaps I am mistaken.

 - Robin

---

From: "Tony Li" <[email-masked]>
Date: Tue, 10 Jun 2008 01:01:19 -0700

Hi Xiaohu

|I wonder whether the RRG is to design a new architecture or 
|just to fix the
|scalability issue of IPv6. And I think it will determine how 
|much change we
|can make in the Internet architecture.
|In other word, should HIP like approach be taken into RRG's 
|consideration
|provided that there was no separate working group and research group in
|IETF/IRTF? 

We have a great deal of freedom to propose anything that results in a better
routing architecture.  Obviously, it should be clearly motivated and a
rational extrapolation from the solution that we set forth.

Could we specifically recommend HIP?  Certainly, however working on it here
would seem somewhat redundant.

Tony

---

From: Xu Xiaohu <[email-masked]>
Date: Tue, 10 Jun 2008 16:14:29 +0800

> rational extrapolation from the solution that we set forth.

Thanks for your clarification.

> would seem somewhat redundant.

I see ;)

Xiaohu XU

---

From: Per Heldal <[email-masked]>
Date: Tue, 10 Jun 2008 11:07:24 +0200

> dual-stack and IPv6-only adoption.

Far from everyone in the ops-community share that view.

Technically, v6 has for some time already had virtually everything v4
has wrt provider and end-user functionality. However, the only really
new feature of v6 is the extra addresses and there's a cost involved to
get started. Hence the lack of interest. 

A safe bet on this planet is that scarcity brings out the worst in
people. There's no reason to assume that IP will be an exception. Many
providers that have a strategy beyond blindly copying others are already
planning for solutions that enable growth through IPv6 at the customer
edge. The widely anticipated steep increase in cost per IPv4 address
will be a powerful catalyst to push investments towards more
future-proof solutions. Regardless of an increased ability to
slice-and-dice v4 address-blocks, nobody has yet presented a realiable
scheme to reclaim enough v4 addresses to maintain anything that
resembles current growth at network's customer edge past depletion.

While IPv4 generally is expected to stay around for a long time there's
also many people in the ops-community who believe that the majority of
the internet may switch to v6 is as little as 3-5 years once it gets
rolling helped by the ever shortening life-span of both software and
hardware. Besides, it is also reasonable to expect a development in
NA(P)T-PT/ALG solutions for v6-only hosts in the next few years that is
similar to the rise of NAT in the late 90s.

Summary: yes we want a new (or supplemental) scalable routing
architecture, but the v4 run-out is highly questionable as an argument
in favour of any particular solution.

//per

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 10 Jun 2008 11:34:53 +0200

Tony Li wrote:
[..]

> would seem somewhat redundant.

You folks have more insight in that than me, but my take on it is that 
we have basically have the following situations:

* Big sites with a big address pool providing connectivity to loads
   of users. These will inject only a IPv6 /32 maybe per country they are
   in. Serve millions of users, require lots of addresses.
   => 1 routing slot per country or so, not a big burden

* Content delivery sites, not a large prefix in general, but actually
   having a large amount of hosts behind them, though only few actually
   will be visible as most are behind load balancers. These might be
   globally distributed and anycasted and might have high routing
   activity for load balancing, data centers going down etc.
   => multiple routing slots, around the world

* End-sites, small places which require static IPs due to firewalling
   and other static configurations (DNS). Don't change routing that
   often serve a 'small' amount of users. Prefix only present at one
   location.
   => 1 slot per small site, with many many many of those sites.

There is of course then also the aspect of Traffic Engineering, which 
for all these above sites can cause extra entries in BGP.

Now, the first one, that is how people envisioned (from what I 
understand that is :) IPv6 routing to happen: large sites giving chunks 
of address space to small sites.

The problem with that of course being that people want to be independent 
and thus 'require' their address space to be globally visible thus we 
get more prefixes in there. Effectively though we only have a couple of 
organizations which actually are real "Tier 1"s, thus really only a few 
organizations really need a slot in the routing tables. Currently IPv6 
allocations are given to anybody who can cough up the dough, this thus 
restricts in effect the really small sites (the latter group). The 
content folks generally have enough cash to cough up for a little prefix 
thus that is not a problem for them.

That actually leaves the latter group who are most likely one of the 
locations that want to have a global independent address but should not 
have a slot in the routing tables if we want to keep those to a low 
number in the long long long run (we really have no idea if that number 
will explode or not, and also of course if the hardware will or will not 
be able to keep up with it, it might just be that they can scale, but 
the question then is can and will everybody scale up?).

Currently we already have two solutions which will solve most of the 
requirements from the latter group: SCTP and HIP. The only big problem 
with both of those being that everybody everywhere needs to start using 
them and have them available, next to changing the mentality of 
application programmers and of course also the people who are developing 
it. Oh and there is of course the issue of people simply stating "we do 
it like this now, lets do it like that tomorrow", which is not helping 
much either.

SCTP would require heavy vendor support to get rolling, especially in 
the Microsoft area it needs to be installed per default, if is not 
installed per default, then (unless there is a magic thing that people 
crave and they will install something for it) SCTP will never happen and 
when it is not at the endsite then it won't happen at the providers 
either. This is more or less the same issue as IPv6: chicken-egg. There 
is one advantage here though, it does not require anything in the 
middle, except maybe firewalls to allow it to work.

Greets,
  Jeroen

---

From: Robin Whittle <[email-masked]>
Date: Tue, 10 Jun 2008 22:32:34 +1000

Hi Per,

Thanks for your response.

I understand there is greater optimism about mass IPv6 adoption
amongst ISPs than I had thought.

Can you point to one or more instances where ordinary end-users are
paying for an IPv6-only service?  Or for a dual protocol IPv4 and
IPv6 service?

Has anyone done a pilot project to see how happy a real sample of
end-users would be relying on NAT-PT or whatever?

NAT-PT systems require some IPv4 space, but not one IP address for
every home customer.  Double IPv4 NAT (NATed address to the DSL
router, which has its own NAT for the customer PCs) has the same
advantage in terms of squeezing more customers onto scarce IPv4
space, without fussing with IPv6 and whatever support costs that
would entail.  However Double NAT won't work with all protocols -
and either can NAT_PT from what I know - so both are probably
unacceptable due to customer dissatisfaction and support costs.

The recent (April 200) review of approaches to NAT-PT:

http://www.arin.net/meetings/minutes/ARIN_XXI/PDF/tuesday/nat-pt-setup.pdf

points to:

  http://www.lucastomicki.net/naptd.docs.php   (Linux)
  http://www.cisco.com/en/US/docs/ios/12_2t/ipv6/SA_natpt.pdf

Linux is somewhat flaky and doesn't scale to large numbers of users.
 Cisco's approach is more stable and scalable, but poorly documented.

This small ID from Comcast:

http://tools.ietf.org/html/draft-durand-v6ops-natv4v6v4-01

discusses the pitfalls of "Double IPv4->IPv4->IPv4 NAT".  Using
"Double IPv4->IPv6->IPv4 NAT" has similar difficulties.

"IPv6 Tunneling plus carrier-grade IPv4->IPv4 NAT" doesn't help,
since "the service provider still needs to assign one IPv4 address
per customer."

That's it for the solution space.  My interpretation of this is that
none of them provide a service which an ordinary user could be happy
with and which an ISP could sell without unacceptable support costs
when some application doesn't work as it should.

It is tricky predicting how IPv4 address space could be used more
effectively, but there is clearly a strong motivation to do so,
since at present and for the foreseeable future, an IPv4 address for
each DSL customer is the only way of providing an Internet service
which meets ordinary end-user's needs without any fuss.

There may not be such a lot of scope for improving utilisation of
IPv4 space in ISP networks, since my impression is they have these
prefixes already chock-a-block with customers.

There is a great deal of scope for map-encap serving the needs of
end-user networks who need multihomeable, portable, space with less
than 256 IP addresses, so reducing the demand for conventional
BGP-managed PI space.

Maybe more applications than I realised are ready for IPv6 - at
least those being currently maintained.  But that isn't much help
for the routing scaling problem until most hosts have IPv6 and there
is no need for most end-users to have an IPv4 address.

There is still a great chicken and egg problem here, and my guess is
that the most likely outcome will be incremental - more and more
intensive use of IPv4 - rather than risky and revolutionary: trying
to sell end-user networks or individual home and office end-users an
IPv6 service which can't meet their need to use all existing
applications without fuss with all existing (IPv4) hosts.

> people. There's no reason to assume that IP will be an exception. 

It may be tempting to think of the IPv4 situation being hopeless so
the masses will soon be forced to jump ship to IPv6.  But I think
this involves getting millions of people to pay for and be happy
with a service which can't do what ordinary IPv4 Internet services
do today.

> Many

There are 3.7 billion IPv4 addresses available.  Probably a few
hundred million of the currently advertised 1.7 billion addresses
are in use:

  http://www.isi.edu/~johnh/PAPERS/Heidemann08a.pdf
  http://www.firstpr.com.au/ip/host-density-per-prefix/

I think that provides plenty of scope for better utilization for
another decade or two, especially with map-encap.  Without
map-encap, there will be finer slicing and more BGP advertised
prefixes - the routing scaling problem heating up.  But still I
think the costs will not be so high as to make it attractive to try
to sell a second-rate problematic service to millions of people in a
competitive environment.  (In China and some other countries,
perhaps there won't be a competitive environment, so IPv6 may well
get its first widespread adoption there.)

> rolling 

Yes, but how will it get rolling?  You need millions of customers to
 adopt IPv6-only services, and be happy with them and pay for them
without lots of support calls.  Otherwise, no-one will develop the
most-used applications to the point where they are reliable and well
documented with IPv6.

How are you going to do that when IPv6 offers no advantage to the
end-user?  IPv6 is a cost-saving thing for the ISP (only to the
extent that an IPv4 address is more expensive), but it doesn't
provide what the customer really wants and needs, which is ordinary
IPv4 connectivity.

That chicken and egg thing wants to happen, but neither the chicken
nor the egg are substantial enough to bring the other into existence.

> helped by the ever shortening life-span of both software and

I would want to see concrete evidence of ordinary end-users being
happy with an IPv6-only service before I could imagine widespread
IPv6 adoption (home, office and corporate end-users and virtually
all web servers etc.), widespread OS and application upgrades and
significant migration from IPv4.

Only then might I be tempted to bet that half or more end-users
might be able to do without their own IPv4 address by 2015 to 2018.

Only then might I be temped to think it would be OK to let IPv4
fester without an architectural upgrade, and think only of working
on IPv6.

  - Robin

---

From: David Conrad <[email-masked]>
Date: Tue, 10 Jun 2008 06:41:05 -0700

Bill,

On Jun 9, 2008, at 8:07 PM, William Herrin wrote:
> The identified problem is:

> prefix in the core.

I'd be curious as to how you derived this number.

> Shifting focus to IPv6 abandons the problem.

Not really, since generally IPv6 routes and IPv4 routes are combined  
when computing routing load.

Regards,
-drc

---

From: "William Herrin" <[email-masked]>
Date: Tue, 10 Jun 2008 10:14:01 -0400

> I'd be curious as to how you derived this number.

Hi David,

http://bill.herrin.us/network/bgpcost.html

The source numbers change with time. Feel free to plug any realistic
source numbers into the formula that you please. The end result will
have the same impact: the cost is nontrivial.

If you want to challenge the methodology, I respectfully ask that you
find someone who does cost analysis for a living to assist you with
the particulars. I had a professional cost analyst review and validate
it. Same guy who got the V22 Osprey canceled before Congress put it
back.

> Not really, since generally IPv6 routes and IPv4 routes are combined when
> computing routing load.

If we do nothing, by what date do you expect IPv6 to exceed even 2% of
the total routing load?

I stand by my statement.

Regards,
Bill Herrin

---

From: David Conrad <[email-masked]>
Date: Tue, 10 Jun 2008 07:23:32 -0700

Robin,

> from the optimal combination of recommendations.

I view the statement proposed by Tony as merely indicating the  
priority is IPv6.  A solution that addresses both IPv6 and IPv4 would  
likely be more interesting than a solution that addresses only one of  
these.

> Despite its length, I think my text has a good signal-to-noise ratio

I do not disagree.  However, if we can't even get a rough consensus on  
28 mostly innocuous words, I'm somewhat skeptical we'd be able to  
reach rough consensus on 443 words that provide many more hooks for  
folks to get hung up on.

> Based on recent messages of support for Tony's text, I am

It may be that given the limitations of IPv4, folks view its  
scalability as a self-correcting problem.  If IPv6 scalability were  
solved, the advantages of having "vast tracts of swamp" ... err...  
lots of address space would, in the end, encourage people to migrate  
away from IPv4 (since they wouldn't be able to get more IPv4 address  
space at a reasonable cost).

> I think we all went to a lot of trouble
> needed ASAP for IPv4.

An interesting question and one for which I have never gotten a  
satisfying answer.  There is pressure for a solution, but it isn't  
clear to me how time critical it is.  Clearly, there is some room to  
maneuver and, in the end, ISPs will take whatever actions they feel  
necessary to protect their own infrastructure, even if that means  
filtering out parts of the Internet.  It would be nice to have a  
solution that precludes the need for such drastic action, but I'm told  
we don't absolutely have to have it today.

> That was the impetus behind the RAWS workshop 20 months ago.

Well, no, not really.  Dave Meyers can correct me if I'm wrong, but I  
believe it was more that the brokenness in routing technology used by  
both IPv4 and IPv6 wasn't getting the attention it deserved.  To  
paraphrase Dave Clark, since (from a routing perspective) IPv6 was  
just a bigger truck driving into the same swamp, we'd be screwed if we  
didn't look at the problem more closely.

> IPv4 than the map-encap systems.

Map-encap obviously isn't limited to IPv4.  In fact, if you're doing  
map-encap, the stuff that you're mapping and encapping is most likely  
an opaque bag of bits relevant only to the stuff on the outside of the  
mapper and encapper.

Regards,
-drc

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 10 Jun 2008 10:44:46 -0400

> it. Same guy who got the V22 Osprey canceled before Congress put it

Note that another hard to estimate number is the economic benefit of  
the Internet, but it
is reasonable to estimate this as at least $ 200 billion / year, but  
probably 2 or 3 times that.
The direct revenue estimates I could find were all > $ 200 billion /  
year.

So, the $ 2 billion / year to run BGP that you estimate is a cost of  
doing business no more than 1% of revenue,
and probably more like 0.3 to 0.5% of Internet revenue, aggregated  
globally.  Non-trivial, to be sure, but that level does  not seem  
unsustainable or even unreasonable.

To put it another way, that prefix is worth the money.

Regards
Marshall

---

From: Jeroen Massar <[email-masked]>
Date: Tue, 10 Jun 2008 17:26:10 +0200

Robin Whittle wrote:
[..]
> paying for an IPv6-only service?  Or for a dual protocol IPv4 and

I have been for the last 4 years or so. Note IPv6 is just an 'addon' as 
the ISP in question (bit.nl) just provides me with full Internet 
connectivity, this thus includes IPv4 and IPv6 over the same link.

Both free.fr and nerim.fr do provide IPv6 on a large scale too, next to 
that: http://www.sixxs.net/faq/connectivity/?faq=native

Check http://www.ipv6style.jp/en/statistics/services/index.shtml for a 
list of Japanese providers doing IPv6+IPv4 over the same link. Also 
there is apparently wide availability in .kr & .tw. In all cases you 
just get "Internet", thus both IPv4 and IPv6 over the same link.

> end-users would be relying on NAT-PT or whatever?

Why are you even thinking of NAT-PT? The above examples generally 
provide 1 IPv4 address (which most people then NAT) and a /48 IPv6.

At one point an ISP will not be able to provide any more public IPv4 
addresses (as it really ran out) to their users they can just start 
re-using those addresses. Eg if they have a /16, they can just cut that 
up into 256 /24's and recycle that /24 various times and use one /24 for 
the real public addresses, thus NATing the rest of the /16 behind that.
Fastweb in Italy has been doing that with a large amount of hijacked 
prefixes (http://plany.fasthosting.it/natpool.html) already for quite 
some while and that seems to work more or less.

Thus leaving the NAT-burden on IPv4 and having full e2e IPv6.

Greets,
  Jeroen

---

From: "William Herrin" <[email-masked]>
Date: Tue, 10 Jun 2008 16:00:26 -0400

> To put it another way, that prefix is worth the money.

Marshall,

Only to the folks who originate the prefix. To everyone else its a
large unrecoverable cost.

Which, by the way, suggests an approach to solving RRG's problem that
we haven't seriously explored: how can we systematically arrange for
the folks who originate a prefix to pay the folks who carry the
prefix?

It's OK for core routers to cost $10M, *IF* that cost can be recovered
from the folks whose routes cause it.

Regards,
Bill Herrin

---

From: David Conrad <[email-masked]>
Date: Tue, 10 Jun 2008 13:33:05 -0700

Bill,

> the folks who originate a prefix to pay the folks who carry the

Long ago, there was an attempt to look into this (IETF PIARA (Pricing  
of Internet Addresses and Routing Announcements) BoF).  If I remember  
correctly, the routing part of it broke down because no one could  
figure out how to get folks to pay (see "Tragedy of the Commons").   
Not sure if things have changed.

It sort of looks a bit like carbon credits...

Regards,
-drc

---

From: "William Herrin" <[email-masked]>
Date: Tue, 10 Jun 2008 18:08:56 -0400

On Tue, Jun 10, 2008 at 4:33 PM, David Conrad <[email-masked]> wrote:
> Long ago, there was an attempt to look into this (IETF PIARA (Pricing of
> Internet Addresses and Routing Announcements) BoF).  If I remember
> correctly, the routing part of it broke down because no one could figure out
> how to get folks to pay (see "Tragedy of the Commons").  Not sure if things
> have changed.

David,

Please excuse my temporary insanity while I step way outside the box.

Set up some sort of central route policy authority, CRPA.

As the AS originating a route, you go to the CRPA and place all other
ASes into one of three categories for EACH route you originate:

Category 1: I will not pay you to carry this prefix.
Category 2: I will pay you $CRPA standard amount per month to carry
this prefix at your published default exit only, if and only if you
arrange your internal routing so that packets to this route will
either reach your default exit (via a default route) or follow a
legitimate externally-generated shorter prefix advertisement if one
exists.
Category 3: I will pay you $Offer total to carry this route on every
BGP router you operate.

Obviously you use templates; few would want or need to explicitly set
rules for each AS.

Each AS who wants to be paid then downloads the list of offers made to
their AS periodically. They load the list into each of their routers
and tell the router to filter accepted prefixes based on  the
following:

a. Carry category 1 prefixes shorter than X.
b. Carry all/no category 2 routes.
c. Carry category 3 routes where $Offer exceeds $Z.
d. All normal prefix filtering (regardless of whether they're in CRPA)

The CRPA acts as a settlement agent as well. Each AS which carries
routes based on the CRPA   provides CRPA with two BGP feeds: one from
the designated default exit and one from the a selected "category 3"
router. CRPA uses this to determine which routes you actually carried
and bills the origin ASes accordingly. When the origins pay CRPA, CRPA
pays you.

Why would anyone start paying?

If you're a transit provider, you can pick up some extra revenue yet
disturb nothing by signing up for CRPA routes and setting your "carry
category 1 shorter than" to carry anything /24 or shorter anyway.
You'll accept longer routes only if paid.

At ARIN and the other RIRs, we go pass public policy allowing end-user
orgs who don't qualify for a block at the current size limits to
register non-contiguous blocks of up to a /25 of addresses. Now a SOHO
can actually get PI addresses if they're willing to pay for the
routing via CRPA.

With such a system in place, it's only a matter of time before folks
start filtering on RIR minimums and/or filtering distant prefixes
based on their presence or absence from CRPA. For example, as a
regional ISP in the Southeast US, I might refuse to carry APNIC
regions prefixes unless their originators pay, shunting them instead
to a transit provider I've picked as my low-cost default exit.

Carrying your routes becomes a paid service, just like any other.

Regards,
Bill Herrin

---

From: David Conrad <[email-masked]>
Date: Tue, 10 Jun 2008 16:13:07 -0700

Bill,

...
> the particulars.

Since I do not have ready access to such an individual, I'll simply  
say that your methodology does not appear to match my intuition or  
what I know about marginal costs.  However, it isn't really relevant  
-- I was just curious how you came up with that number.

> If we do nothing, by what date do you expect IPv6 to exceed even 2% of

Invalid measure since the theory is that IPv6 won't be as deaggregated  
unnecessarily as IPv4.  Of course, I don't really believe that theory,  
but that's a separate topic.  A more reasonable measure would likely  
be at what date to I expect IPv6-reachable sites to exceed even 2% of  
the IPv4 reachable sites.  I'd pick a number out of the air and say  
something like 5 years after RIR IPv4 runout.

However, I'd argue this isn't really relevant either. As I said in my  
note to Robin (ignoring all the fanciful scenarios), if we make IPv4  
routing scale better, all it would seem to do would be to promote  
deployment of more NAT.  I'm not sure this is all that a desirable  
longer term goal (which is my understanding of RRG efforts) -- you'd  
end up with a universe of NATv4 endpoints, possibly even flat routed  
down to /32s (which might even be feasible by the time it became an  
issue).

Pragmatically speaking, I believe RRG should be looking at a desirable  
future: one that gets us out of the scaling hole while at the same  
time allowing for end point identifier stability in the face of  
network topology changes.  Since I think we've already demonstrated  
IPv4 can't provide end point identifier stability (even if we could  
make the scaling hole magically go away), it seems to me we should be  
focused on a solution that we know can (if not will) provide for end  
point identifier stability.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Tue, 10 Jun 2008 16:24:30 -0700

Bill,

> Please excuse my temporary insanity while I step way outside the box.

Don't worry, been there myself.  The view is nice, but watch that  
first step... :-)

> Set up some sort of central route policy authority, CRPA.

"First, boil the ocean."  :-)

As I responded to someone else privately, back when this was discussed  
in PIARA and CIDRD, the sticking point was generally along the lines  
of none of the big players were interested in jumping into something  
that required cooperation with competitors when they could just as  
easily spend money to upgrade their routers (with the possibility that  
their competitors couldn't and thus would cease to be competitors).   
To get something like this to work, you need to have some sort of  
enforcement mechanism, which generally implies governmental entities.   
One could argue that the RIRs could be the enforcer, but no one (at  
the time) felt it a good idea to give the RIRs that much power over  
routing.

I don't think things have changed all that much in this respect  
(although perhaps with the RPKI stuff...)

> based on their presence or absence from CRPA.

Without the CRPA, you'll get this anyway as ISPs act to protect their  
own infrastructure.  I'm not sure this is a desirable outcome.  You  
can either make people pay for their use of the commons or figure out  
how to reduce the impact of use of the commons.  Routing settlements  
is the former.  I believe a better choice going forward is the latter.

Regards,
-drc

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jun 2008 11:52:29 +1200

On 2008-06-11 02:44, Marshall Eubanks wrote:
> Note that another hard to estimate number is the economic benefit of the
> Internet, but it
> The direct revenue estimates I could find were all > $ 200 billion / year.
> 

Of course it is, or the capitalist nature of the Internet would have
caused that prefix to be filtered out. There is an incentive for
all ISPs to share the externality costs of BGP advertisements from
other ISPs - the fact that connectivity is its own reward.

But common sense suggests that the externality cost *per customer*
of an aggregated prefix is lower than for a deaggregated prefix.
So I think Bill's average rate of $8k/prefix (whether it's correct
or not) isn't very helpful, because it conceals the different values
of aggregated and deaggregated prefixes. We need to look at
cost per customer.

    Brian