Subject: [RRG] IPv4: how bad, when to fix?

From: Robin Whittle <[email-masked]>
Date: Wed, 25 Jun 2008 11:28:44 +1000

Can we agree that we need to recommend a fix for the IPv4 routing
scaling problem - and that this is our most urgent priority?  I
propose some text for potential consensus in a separate message
"Take 2: AA, BB, CC".

Here are some arguments about why we need to fix IPv4 first.

In the thread: "Re: [RRG] Long term clean-slate only for the RRG?"
Yakov Rekhter wrote:

> Perhaps we should first agree that there is a need a *short term*
> solution for both IPv4 and IPv6. The following (from Tony's e-mail
> on 5/26/2008) is relevant to the discussion on whether there is
> such a need:
>
>   Well, Ross Callon has been quoted as saying that the Juniper
>   implementation will have no problems up through many millions
>   of routes.
>
>   Now, conceptually, that could happen tomorrow.  However, at the
>   current growth rates, that's likely to be many years.

Previously, in http://psg.com/lists/rrg/2008/msg01612.html, Brian
Carpenter wrote:

> Dino Farinacci wrote:
> ...
>> Could we agree that one map-n-encap solution for both IPv4 and
>> IPv6 be a short-term solution while we work on a long-term
>> solution for IPv6-only?
>
> Clarifying question: do we have to hold our breath while
> waiting for the long term solution?
>
> But seriously - since it doesn't seem that we want to shrug our
> shoulders over IPv4, and I don't see the industry being willing
> to support two solutions, I don't see much choice.

I agree we can't shrug our shoulders and let today's Internet
degenerate, solely on the basis that we are devising another
Internet to which we assume everyone will happily migrate before too
long.

I would much rather develop a long-term solution at a relaxed pace
without expecting the rest of the world to hold its breath and wait
for our great creation to be ready for mass inhabitation.

Some other perspectives which seem to be critical of the view that
router technology will cope acceptably with the IPv4 routing scaling
problem - so we don't need to fix it and should concentrate on an
IPv6 or clean-slate solution:

In a message which was addressed to the list, but has not so far
been propagated, Eliot Lear wrote:

> Yakov,
>
> If IPv6 is any measure it could take 15 years to deploy the next
> thing.  Are you are willing to categorically state that there is
> no problem for that period of time?

Dino wrote:

> The routing table size problem is not the only problem. There are
> many enterprise sites that want to do low-cost multihoming, they
> want to be good citizens to the Internet and don't want to inject
> more specifics, and they want to control their ingress traffic
> flows.

Regarding the quote from Russ Callon: it is no surprise people from
the big router vendors say they can build routers to handle "many
millions of routes".  I am sure they can, for a price.  They would
be averse to stating there was some limit to their technical
capability.  However, Tony Li and maybe others in the RAWS workshop
did point out that relying on Moore's Law to cope with the scaling
problem was unrealistic.

http://www.iab.org/about/workshops/routingandaddressing/Router_Scalability.pdf

One difficulty with throwing silicon at the problem is that at some
point, the job has to be split between multiple CPUs with their own
RAM.  This raises some really nasty problems with costs, software
complexity and the CPU resources devoted to those separate route
processor systems communicating with each other so they behave as
one.  Likewise, fail-over protection becomes trickier still.

There are several other problems with this "millions of routes" idea.

Firstly, there are stability concerns with the BGP system handling
such numbers of advertised prefixes.

Secondly the number of routes a DFZ router can handle depends very
much on its memory and CPU capacity, in terms of how it handles the
number of DFZ routes *multiplied* by the number of interfaces which
 link to other DFZ routers.

In broad principle, the router has to maintain a separate
"conversation" about each route with each DFZ neighbour router - and
so has to engage in communications, store state etc. in proportion
to the total number of such "conversations".

So the adequacy of a "millions of routes" router in a particular
location depends just as critically on how many DFZ neighbours it
has as on how many routes there are in the DFZ.

   (It has been hard to determine how many DFZ routers there are,
    but a figure of 210k is regarded as a minimum.  Also, there is
    limited information about the distribution of the numbers of
    neighbours.  See the "Routers in DFZ - reliable figures from
    iPlane" thread:

       http://psg.com/lists/rrg/2007/msg00253.html
       http://psg.com/lists/rrg/2007/msg00255.html
       http://psg.com/lists/rrg/2007/msg00257.html
       http://psg.com/lists/rrg/2007/msg00262.html )

Thirdly, these mega routers are going to cost significantly more
than routers with CPU and RAM resources suitable for current (~260k)
numbers of DFZ routes.

Depending on the number of DFZ neighbours for any given router, and
the growth in the number of DFZ routes in the years to come, all DFZ
routers may need to be replaced with these expensive things.  To the
extent that these more expensive replacements occur partially or
solely due to the growth in the number of DFZ routes, then this is a
direct cost burden on ISPs and transit providers which could be
avoided by a solution for IPv4.  These costs are ultimately paid by
all Internet users.  (The trick is to devise a scalability solution
where the costs of adoption are less than the immediate benefits, so
it will be widely adopted.)

Since such costs of mega routers depend in part on the number of DFZ
routes, unconstrained growth in this number will cause increasingly
strong pressure against the further splitting up of the IPv4 address
space via the current BGP-only address management system.  This
further increases the costs and difficulties faced by the growing
number of end-user networks which need, but can't get, portable,
multihomable space.

The above seems to be restating the themes of RAWS 1.66 years ago:

  http://tools.ietf.org/html/rfc4984
  http://www.iab.org/about/workshops/routingandaddressing/

which itself largely involved restating concerns which have been
known for 16 or so years earlier.

All along this process, as far as I can see, some folks have been
focusing on fixing the IPv4 routing scaling problem first, while
others have been thinking it would be OK not to worry about it, and
to get IPv6 ship-shape in time for the mass adoption which has
always been expected "in the next few years".

Now, some people foresee IPv4 becoming rapidly more unworkable due
to the address depletion problem, which buoys their hopes for the
long-awaited mass-migration to IPv6.  I have argued there is a lot
of scope, especially with map-encap, for extending the productive
life of IPv4:

  Many more end-user networks could get their own portable,
  multihomable prefixes via map-encap, with little burden on the
  number of DFZ prefixes.  This is true as long as the DFZ prefixes
  which are the "Mapped Address Blocks" (Ivip terminology) the
  map-encap system splits into micronets (Ivip) or EID prefixes
  (LISP) are generally big blocks split which are split into many
  smaller blocks for large numbers end-user networks.

Opinions vary widely on how realistic this scenario is.  It depends
a lot on our visions of end-user networks, how small a patch of
portable, multihomable, public space they would be happy with, how
we think IPv4 space could be managed with map-encap, how address
space would in fact be managed as the address crunch develops etc.
Most people don't foresee much scope for improving IPv4 utilization,
but I think there will be enormous pressure to do this, and various
methods, including map-encap, will be devised to achieve it.

We have 0.75 years before the RRG's deadline.

We are supposedly going to produce a "single recommendation", but I
don't see how this is realistic.

It would take years to decide reliably on a long-term solution -
probably involving a radically new architecture, protocols and
therefore stack and applications.

I think we may be able to devise an optimal solution for IPv4.

Maybe we could do the same for IPv6 too.  However, that is not so
urgent.  Yet we want to have some idea about potential IPv6 or clean
slate solutions to help us make the IPv4 solution as compatible as
possible with the long-term solution.

I think there is something more important than theoretical
discussions on commonalities between architectures, such as Lixia's
focus: http://psg.com/lists/rrg/2008/msg01599.html

   >> The most substantial solutions to the routing scaling problem
   >> proposed so far are:
   >>
   >>    LISP-NERD
   >>    LISP-ALT
   >>    APT
   >>    Ivip
   >>    TRRP
   >>    Six/One Router
   >
   > From architectural view, I'd identify the commonality of
   > all the above as stopping topology-aggregatable prefixes at
   > the boundary of edge sites, not which version of IP they
   > may/not work with.

While this is true, Six/One Router can't solve the IPv4 scaling problem.

Since I think we need to work fast on IPv4 and can take more time
with IPv6 and clean-slate solutions, the question of whether an
architectural enhancement can support IPv4 is of primary importance
in our most urgent work.  In other words, we can't afford to ignore
important "details" such as the differences between IPv4 and IPv6,
and which potential solutions are applicable to them.

IPv4 is the only current problem.  Unless we are very relaxed about
continuing shortage of multihomable space for end-user networks, the
pressure this puts on the scaling problem when using BGP-managed PI
space for this purpose, the costs this places on ISPs, and the
difficulties faced by the end-user networks who can't get the space
they need, then this is a problem which needs to be fixed in the
next 5 to 8 years.

We can't reliably expect to have an alternative, scalable, Internet
widely adopted anywhere near soon enough to bring the IPv4 problem
to a halt in an acceptably short time.

Supporters of the contrary line - that we don't need to fix IPv4, or
that we can maybe fix it after we have figured out an IPv6 or
clean-slate solution - need to argue convincingly about development
and deployment times for their non-IPv4 solutions.  Then they need
to argue convincingly why most end-users would adopt new
applications, new IPv6 (or whatever) Internet services, and
reconfigure their entire networks and IT systems (to dual stack
initially), in a sufficient short time that the IPv4 scaling problem
would be acceptably limited.

  - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "Marcus Brunner" <[email-masked]>
Date: Wed, 25 Jun 2008 11:37:35 +0200

BTW, we might also want to have consensus on what should not be changed, what part o fhte architecture should stay as is and has proven to work.

Kind regards,

Marcus

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf 
> Of Robin Whittle
> Sent: Wednesday, June 25, 2008 3:29 AM
> To: Routing Research Group
> Subject: [RRG] IPv4: how bad, when to fix? 
> 
> Can we agree that we need to recommend a fix for the IPv4 
> routing scaling problem - and that this is our most urgent 
> priority?  I propose some text for potential consensus in a 
> separate message "Take 2: AA, BB, CC".
> 
> > solution for both IPv4 and IPv6. The following (from Tony's 
> e-mail on 
> > 5/26/2008) is relevant to the discussion on whether there is such a 
> > need:
> >
> Previously, in http://psg.com/lists/rrg/2008/msg01612.html, 
> Brian Carpenter wrote:
> 
> > Dino Farinacci wrote:
> > ...
> solution 
> >> for IPv6-only?
> > shoulders over IPv4, and I don't see the industry being willing to 
> > support two solutions, I don't see much choice.
> 
> Internet to which we assume everyone will happily migrate 
> before too long.
> 
> I would much rather develop a long-term solution at a relaxed 
> pace without expecting the rest of the world to hold its 
> breath and wait for our great creation to be ready for mass 
> inhabitation.
> 
> Some other perspectives which seem to be critical of the view 
> that router technology will cope acceptably with the IPv4 
> routing scaling problem - so we don't need to fix it and 
> should concentrate on an
> In a message which was addressed to the list, but has not so 
> far been propagated, Eliot Lear wrote:
> 
> > Yakov,
> >
> > thing.  Are you are willing to categorically state that there is no 
> > problem for that period of time?
> 
> 
> Dino wrote:
> 
> > The routing table size problem is not the only problem. 
> There are many 
> > enterprise sites that want to do low-cost multihoming, they 
> want to be 
> > good citizens to the Internet and don't want to inject more 
> specifics, 
> > and they want to control their ingress traffic flows.
> 
> 
> Regarding the quote from Russ Callon: it is no surprise 
> people from the big router vendors say they can build routers 
> to handle "many millions of routes".  I am sure they can, for 
> a price.  They would be averse to stating there was some 
> limit to their technical capability.  However, Tony Li and 
> maybe others in the RAWS workshop did point out that relying 
> on Moore's Law to cope with the scaling problem was unrealistic.
> 
> One difficulty with throwing silicon at the problem is that 
> at some point, the job has to be split between multiple CPUs 
> with their own RAM.  This raises some really nasty problems 
> with costs, software complexity and the CPU resources devoted 
> to those separate route processor systems communicating with 
> each other so they behave as one.  Likewise, fail-over 
> protection becomes trickier still.
> 
> Firstly, there are stability concerns with the BGP system 
> handling such numbers of advertised prefixes.
> 
> Secondly the number of routes a DFZ router can handle depends 
> very much on its memory and CPU capacity, in terms of how it 
> handles the number of DFZ routes *multiplied* by the number 
> of interfaces which  link to other DFZ routers.
> 
> "conversation" about each route with each DFZ neighbour 
> router - and so has to engage in communications, store state 
> etc. in proportion to the total number of such "conversations".
> 
> So the adequacy of a "millions of routes" router in a 
> particular location depends just as critically on how many 
> DFZ neighbours it has as on how many routes there are in the DFZ.
> 
> Thirdly, these mega routers are going to cost significantly 
> more than routers with CPU and RAM resources suitable for 
> current (~260k) numbers of DFZ routes.
> 
> Depending on the number of DFZ neighbours for any given 
> router, and the growth in the number of DFZ routes in the 
> years to come, all DFZ routers may need to be replaced with 
> these expensive things.  To the extent that these more 
> expensive replacements occur partially or solely due to the 
> growth in the number of DFZ routes, then this is a direct 
> cost burden on ISPs and transit providers which could be 
> avoided by a solution for IPv4.  These costs are ultimately 
> paid by all Internet users.  (The trick is to devise a 
> scalability solution where the costs of adoption are less 
> than the immediate benefits, so it will be widely adopted.)
> 
> Since such costs of mega routers depend in part on the number 
> of DFZ routes, unconstrained growth in this number will cause 
> increasingly strong pressure against the further splitting up 
> of the IPv4 address space via the current BGP-only address 
> management system.  This further increases the costs and 
> difficulties faced by the growing number of end-user networks 
> which need, but can't get, portable, multihomable space.
> 
> which itself largely involved restating concerns which have 
> been known for 16 or so years earlier.
> 
> All along this process, as far as I can see, some folks have 
> been focusing on fixing the IPv4 routing scaling problem 
> first, while others have been thinking it would be OK not to 
> worry about it, and to get IPv6 ship-shape in time for the 
> mass adoption which has always been expected "in the next few years".
> 
> Now, some people foresee IPv4 becoming rapidly more 
> unworkable due to the address depletion problem, which buoys 
> their hopes for the long-awaited mass-migration to IPv6.  I 
> have argued there is a lot of scope, especially with 
> map-encap, for extending the productive life of IPv4:
> 
> Opinions vary widely on how realistic this scenario is.  It 
> depends a lot on our visions of end-user networks, how small 
> a patch of portable, multihomable, public space they would be 
> happy with, how we think IPv4 space could be managed with 
> map-encap, how address space would in fact be managed as the 
> address crunch develops etc.
> Most people don't foresee much scope for improving IPv4 
> utilization, but I think there will be enormous pressure to 
> do this, and various methods, including map-encap, will be 
> devised to achieve it.
> 
> 
> We are supposedly going to produce a "single recommendation", 
> but I don't see how this is realistic.
> 
> It would take years to decide reliably on a long-term 
> solution - probably involving a radically new architecture, 
> protocols and therefore stack and applications.
> 
> Maybe we could do the same for IPv6 too.  However, that is 
> not so urgent.  Yet we want to have some idea about potential 
> IPv6 or clean slate solutions to help us make the IPv4 
> solution as compatible as possible with the long-term solution.
> 
> 
> Since I think we need to work fast on IPv4 and can take more 
> time with IPv6 and clean-slate solutions, the question of 
> whether an architectural enhancement can support IPv4 is of 
> primary importance in our most urgent work.  In other words, 
> we can't afford to ignore important "details" such as the 
> differences between IPv4 and IPv6, and which potential 
> solutions are applicable to them.
> 
> IPv4 is the only current problem.  Unless we are very relaxed 
> about continuing shortage of multihomable space for end-user 
> networks, the pressure this puts on the scaling problem when 
> using BGP-managed PI space for this purpose, the costs this 
> places on ISPs, and the difficulties faced by the end-user 
> networks who can't get the space they need, then this is a 
> problem which needs to be fixed in the next 5 to 8 years.
> 
> We can't reliably expect to have an alternative, scalable, 
> Internet widely adopted anywhere near soon enough to bring 
> the IPv4 problem to a halt in an acceptably short time.
> 
> Supporters of the contrary line - that we don't need to fix 
> IPv4, or that we can maybe fix it after we have figured out 
> an IPv6 or clean-slate solution - need to argue convincingly 
> about development and deployment times for their non-IPv4 
> solutions.  Then they need to argue convincingly why most 
> end-users would adopt new applications, new IPv6 (or 
> whatever) Internet services, and reconfigure their entire 
> networks and IT systems (to dual stack initially), in a 
> sufficient short time that the IPv4 scaling problem would be 
> acceptably limited.
> 
>   - Robin
> 
> 
> --

---

From: [email-masked] (Noel Chiappa)
Date: Wed, 25 Jun 2008 11:18:20 -0400 (EDT)

> From: "Marcus Brunner" <[email-masked]>

    > we might also want to have consensus on what should not be changed,
    > what part of the architecture should stay as is and has proven to work.

Interesting point.

I think about all everyone agrees needs to be changed is the use of a single
name(space) for both location and identity. Past that, e.g the details of how
the path selection should work, I'm not sure there is much agreement. For
path selection, I suspect it's what Dave Clark called a tussle space: the
users and the providers probably have different ideas of how much control
various entities should have over the selection of the path.

As far as what people agree needs to be kept, I suspect that about all
everyone agrees ought to be kept is use of unreliable packets, and the use of
sequence-numbers/checksums/retransmissions to build reliable end-end
communication on top of them. We may wind up keeping a lot more - a great deal
more, in fact - because of practical reasons associated with deployed base
(e.g. replacing TCP isn't feasible), but that's different.

	Noel