Subject: Re: [rrg] BGP scaling limit?

From: Steven Blake <[email-masked]>
Date: Thu, 01 Jan 2009 11:47:17 -0500

On Thu, 2009-01-01 at 14:59 +1100, Robin Whittle wrote:

> Short version:  Limits on using PC hardware for router FIBs.
>                 Fast SRAM is needed, not DRAM.

[snip]

One can build a 4M+ prefix, 300M LPM/sec FIB engine (v4 or v6) using
commercially available FPGA + DDR3.

Regarding RIB processing: I'll accept the argument that rate*state
growth is exceeding Moore's Law.  However, most RPs I've seen are anemic
compared to commodity server hardware (for legitimate reasons).  So
there is plenty of room for improvement here.

Regards,

// Steve

_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "Tony Li" <[email-masked]>
Date: Thu, 1 Jan 2009 16:33:00 -0800

Hi Paul, 

|ICBW, but I think you're describing what's been called the "Memory 
|wall" by Wulf & McKee:
|
|  http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf
|
|and see also:
|
|  http://www.csl.cornell.edu/~sam/papers/cf04.pdf

Thank you for the excellent references, I think they're exactly on point.
Routing is simply one of the applications that is going to hit the memory
wall.

Tony

---

From: Ross Callon <[email-masked]>
Date: Fri, 2 Jan 2009 11:30:42 -0500

Early on in this discussion engineers from multiple router vendors
stated publicly that that their engineers knew how to build routers that
would handle FIBs that were at least 4M entries or more (10M was
mentioned), using commodity hardware parts. It also became clear, at
least to a few of us talking on the side, that what the hardware guys
were telling one router vendor, in terms of what memory parts they will
have in a few years, is the same thing that they are telling other
router vendors. In various private conversations it also became clear to
me that the router hardware engineers working on the hardware forwarding
parts of routers were not even considering any major change to the
architecture, for the simple reason that they didn't need to.

There may be many reasons to consider making a major change to the way
that the Internet does routing and addressing, but the inability to
build equipment that can handle several million entries is NOT a valid
reason. 

Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Steven Blake
Sent: 01 January 2009 11:47
To: Robin Whittle
Cc: RRG
Subject: Re: [rrg] BGP scaling limit?

[snip]

Regards,

// Steve

---

From: "William Herrin" <[email-masked]>
Date: Fri, 2 Jan 2009 14:13:45 -0500

On Wed, Dec 31, 2008 at 6:36 PM, Christopher Morrow
<[email-masked]> wrote:
>> Sure I can. My $2k COTS PC can route 500mbps easily. Packet switching
>
> sorry, of course you can make the assumption, I just don't think it
> follows as closely to reality as you'd want. that 500mbps isn't doing
> anything tough, with a small table, few interfaces, no services, no
> acls/filtering/qos ... consumer PC's just aren't a good correlary for
> backbone devices.

Hi Chris,

My hypothetical $2k PC is grounded in the recent NANOG discussion
entitled "gigabit Linux routers." The original poster in that
discussion specifically wanted to do BGP, filtering and shaping under
Linux. The NANOG discussion concluded that such a construct would
reasonably route between 700mbps and 1gbs on current hardware. It was
criticized as being less user-friendly and still buggier than a
purpose-built router.

So, according to the Ops, my $2k PC is validly comparable to a modern
core router at the 500mbps level, with plenty of headroom.

>> If you can move packets across an interface at the given data rate
>> then you can blindly multiplex and demultiplex packets from and to
>> lower speed interfaces. This reduces the problem to one previously
>> solved. There's doubtless a better solution but for conceptual
>> purposes, that eliminates the data rate boundary condition as an
>> active factor in the scalability assessment.
>
> at a gross level it might be comparable, but I'm not sure it
> helps cost a device that'll fit in a half-rack and push 1600gbps
> (T1600-juniper/ASR9k-cisco).

I'm not looking for that cost. If I can demonstrate that there is an
upper-bound cost for building a router whose cost varies linearly with
the gbps pushed, without regard to the BGP table size then I can
eliminate the router's gbps rate from the equation that describes the
upper bound for processing BGP.

This says nothing about the average case, in which I can build a
cheaper router whose cost does vary with the BGP table size. But I
don't need to find BGP's average case to calculate an upper bound. And
I certainly don't need to find the average case to demonstrate that a
calculable upper bound exists.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004
_______________________________________________
rrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/rrg

---

From: "William Herrin" <[email-masked]>
Date: Fri, 2 Jan 2009 15:19:35 -0500

On Wed, Dec 31, 2008 at 7:00 PM, Tony Li <[email-masked]> wrote:
> |Do the folks on this group clearly understand the harm that the
> |Regional Internet Registries do by denying number resources to
> |multihomed registrants based on our assurance that it's a necessary
> |evil in order to keep BGP stable?
>
> AFAIK, the RIRs are, for v4, requiring sane and reasonable justification for
> PI prefixes.  For v6, they're effectively giving everyone PI.  Can you be
> more specific about the policies that you're objecting to.

Tony,

ARIN policy requires multihomed users to demonstrate a need to deploy
500 machines on the Internet in order to qualify for end-use IPv4
address assignments. This number was selected above the operations
minimum of 125 addresses based on the IETF and NANOG advice that their
assistance is required to conserve routing slots so as not to overload
the BGP system.

ARIN policy requires the same 500-machine demonstration for multihomed
users to qualify for IPv6 end-user address assignments. This is based
solely on the IETF and NANOG advice that their assistance is required
to conserve routing slots so as not to overload the BGP system.

Entities unable to demonstrate a need to multihome 500 machines are
refused direct assignments, resulting in tying to a single ISP and
incomplete propagation of multihoming addresses within BGP.

> |If we still can't reliably quantify a BGP scaling limit then our
> |request that the RIRs suppress BGP growth by suppressing resource
> |assignment is simply unconscionable.
>
> Do we need to reliably quantify where the Earth will reach the tipping point
> into a irreversible greenhouse effect before global warming is relevant?

Yes, we do.

Geology suggests that absent human behavior, the Earth's current trend
should be towards an ice age. This is a result we'd also like to
avoid. Until we clearly understand the growth pattern associated with
human induced climate change and that pattern's interaction with the
natural pattern, we won't be able to predict the overall pattern with
any accuracy and thus can't rule out the possibility that we need to
increase greenhouse emissions to avoid the next cycle of glaciation.

The 100-year projections for global temperatures are so much junk
science. We still only have a 50/50 chance of predicting the weather
next week.

Regards,
Bill Herrin

---

From: "William Herrin" <[email-masked]>
Date: Fri, 2 Jan 2009 15:20:53 -0500

On Fri, Jan 2, 2009 at 11:30 AM, Ross Callon <[email-masked]> wrote:
> mentioned), using commodity hardware parts. It also became clear to

Ross,

So, effective 2008 we can build routers that handle 4M routes.

Effective 1995, we scrambled to deploy CIDR so that among other things
we wouldn't hit the wall on our state of the art Cisco 7000 routers
which IIRC could handle about 50k routes, that by stretching
then-available hardware to its limits.

Over 13 intervening years, that's somewhere between a compound rate of
improvement of 40% per year and a fixed rate of improvement of 300k
entries per year.

If you fit a growth curve to http://bgp.potaroo.net/ , the BGP table
size is growing at a compound rate of roughly 25% per year.

If our ability to build routers is indeed compounding at 40% per year,
the BGP table size will never catch it.

If our ability to build routers is growing linearly by only 300k
entries per year, the BGP table size will finally catch up with it in
2024 when the routing table has 8.5M entries.

Yes?

Regards,
Bill Herrin

---

From: "Tony Li" <[email-masked]>
Date: Fri, 2 Jan 2009 12:34:22 -0800

Hi Bill,
 
|Entities unable to demonstrate a need to multihome 500 machines are
|refused direct assignments, resulting in tying to a single ISP and
|incomplete propagation of multihoming addresses within BGP.

Seems not unreasonable to me.  Since our current architecture can't support
that kind of multi-homing load, we're trying to avoid the load.

If you're arguing that the status-quo doesn't meet users needs, I agree with
you, of course.  If you're arguing that we should do something to meet those
needs, I also agree.  However, in the case where we fail to do anything,
yes, those needs will be unmet.  This is the obvious outcome unless we can
overcome the partisan bickering and find compromise.

|> Do we need to reliably quantify where the Earth will reach 
|the tipping point
|> into a irreversible greenhouse effect before global warming 
|is relevant?
|
|Yes, we do.

Cool.  Enjoy your Hummer.

Why are you spending your time here then?

Tony

---

From: "Tony Li" <[email-masked]>
Date: Fri, 2 Jan 2009 12:39:54 -0800

Hi Ross,

|There may be many reasons to consider making a major change to the way
|that the Internet does routing and addressing, but the inability to
|build equipment that can handle several million entries is NOT a valid
|reason. 

Please argue on point.

The argument is not about the absolute value of the number of routes
supported.  We can always scale things by throwing more money at gear.

The argument is that the rate of growth exceeds the constant cost curve
delivered by technological improvements, thus resulting in increasing costs
over time.  

Please understand the distinction.

Regards,
Tony

---

From: "William Herrin" <[email-masked]>
Date: Fri, 2 Jan 2009 16:34:20 -0500

Tony,

Now wait a minute; Brian just told us that we can't say with any
scientific or engineering certainty what kind of multihoming load the
current architecture can or can't support. Which is it? Can we? Or
can't we?

You follow the logical inconsistency, right? You can't in one breath
say that we can't determine X and then in the next breath say that
because of X we can confidently say Y.

> Cool.  Enjoy your Hummer.

Mustang GT I'll have you know. Bought it for dirt right before the
price of gas started its nose dive. My 20 mile commute includes a 15
mile stretch of nearly empty freeway. I'm lovin' it!

Regards,
Bill Herrin

---

From: "Christopher Morrow" <[email-masked]>
Date: Fri, 2 Jan 2009 16:58:10 -0500

> assistance is required to conserve routing slots so as not to overload

I believe actually if you can demonstrate a 'unique routing policy'
you can at least get a /24 from one provider, and an ASN from
ARIN/RIPE/APNIC and enjoy a routing slot in the DFZ all your own. When
I was last doing allocation-policy-bits for an ISP we regularly
pointed out to customers that tying themselves to our ip space might
not be in their best interest, and that it really wasn't all that hard
to show usage such that a /22 was a viable option from their local
registry.

-Chris

---

From: "William Herrin" <[email-masked]>
Date: Fri, 2 Jan 2009 17:15:30 -0500

On Fri, Jan 2, 2009 at 4:58 PM, Christopher Morrow
<[email-masked]> wrote:
> ARIN/RIPE/APNIC and enjoy a routing slot in the DFZ all your own.

Hi Chris,

Correct. Your ISP can justify giving you a /24 in their next request
if you request the /24 for multihoming purposes. And ARIN will then
give you an AS#. However, that /24 will often come from a /8 where
ARIN has declared the minimum allocation size to be /20. Though
fortunately few, some carriers treat all /24 cutouts in such blocks as
traffic engineering and filter them out. This results in an incomplete
multihoming solution which still consumes a slot in most folks'
routing table.

I'd be lying if I said I thought the policy was internally consistent.
Nevertheless, it exists and it springs from the assurance by the IETF
and NANOG that we really must suppress multihoming or else BGP will
overload and collapse.

Regards,
Bill Herrin

---

From: "Christopher Morrow" <[email-masked]>
Date: Fri, 2 Jan 2009 20:35:38 -0500

> ARIN has declared the minimum allocation size to be /20. Though

that really depends on the ISP, their allocations and the person(s)
doing the allocation at the time...There was no distinction like that
where I was recently doing this.

> traffic engineering and filter them out. This results in an incomplete

sure, there are examples of this, mostly those folks are limiting
prefixes for financial reasons (don't want to upgrade 7500's or ...)

> multihoming solution which still consumes a slot in most folks'

also true.  My point, and I realized as I hit send I was splitting
hairs, was that routing slot allocations come from a range of places,
just noting that RIR's don't allocation without (hopefully) just cause
isn't the only way a slot is won.

Also, one point that I feel needs to be addressed is that internal
route tables are most often quite a bit larger than the globally
visible table. Ted and I both made this point at the RAWS meeting in
AMS (and it's in the slides I pointed to previously). That growth is
not as easy to quantify from the outside, unfortunately, and also not
well documented.

-Chris

---

From: "Tony Li" <[email-masked]>
Date: Sat, 3 Jan 2009 04:31:02 -0800

|> Seems not unreasonable to me.  Since our current 
|architecture can't support
|> that kind of multi-homing load, we're trying to avoid the load.
|
|Now wait a minute; Brian just told us that we can't say with any
|scientific or engineering certainty what kind of multihoming load the
|current architecture can or can't support. Which is it? Can we? Or
|can't we?

We already know that we're growing the table faster than technology can keep
up.  That much is clear.

|You follow the logical inconsistency, right?

If you're expecting Brian and I to be consistent, I have a bridge to sell
you.  ;-)

Tony

---

From: Ross Callon <[email-masked]>
Date: Sat, 3 Jan 2009 12:47:34 -0500

> We already know that we're growing the table faster than technology
> can keep up.  That much is clear. (Tony Li)

Tony;

I have a question and a comment.

First the question: What do you mean by "we're growing the table faster
than technology can keep up". Are you referring to Forwarding Table
size, BGP processing and traffic loads, or something else? Do you mean
"with a table of 100,000,000 entries, there will need to be at a minimum
some incremental update to existing deployed implementations" (to which
my teenage daughter might say "duh!"), or do you mean "it won't be
economically reasonable to build products unless we make some localized
internal (within a router) change to a few algorithms?", or do you mean
"it will be financially too expensive to build products that can handle
some future table size", or do you mean "it will not be technically
possible to build the routers", or do you mean something else? 

Secondly, depending upon precisely what you mean by "faster than the
technology can keep up", speaking as an individual, it is quite likely
that I disagree. I hope that any disagreement is not automatically
declared to be off-topic. 

Ross

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 04 Jan 2009 08:32:18 +1300

> If you're expecting Brian and I to be consistent, I have a bridge to sell

I will expect commission on that deal. However, the fact that the esteemed
Mr Li and the esteemed Mr Callon appear to disagree supports my assertion
that we don't know when we'll hit the wall. We do know it's two years
closer than it was at the RAWS workshop, though, so the conclusion that
we need a solution in hand stands, IMHO.

   Brian

---

From: "William Herrin" <[email-masked]>
Date: Sat, 3 Jan 2009 16:20:18 -0500

Tony,

Is that a falsifiable scientific theory or is it a consensus opinion
without quantitative grounding?

Devil's advocate:

The well-grounded trend lines I posted suggest that the BGP table size
at the edge of what we can build cost-effectively may be compounding
at a rate of 40% per year while the actual size of the BGP table may
only be compounding at around 25% per year. If true, one consequence
should be that the minimum cost of an edge device capable of
participating in the BGP backbone has fallen significantly. This
appears to be the case; a $1400 Cisco 2811 is now capable of
interacting with the full BGP table while 15 years ago you had to
spend $60k on a Cisco 7000 to seriously think about a full BGP table.

So no, the truth of the claim that we're growing the table faster than
technology can keep up is not at all clear.

Regards,
Bill Herrin

---

From: "Tony Li" <[email-masked]>
Date: Sat, 3 Jan 2009 13:38:43 -0800

Hi Ross,

|> We already know that we're growing the table faster than technology
|> can keep up.  That much is clear. (Tony Li)
|
|First the question: What do you mean by "we're growing the table faster
|than technology can keep up". Are you referring to Forwarding Table
|size, BGP processing and traffic loads, or something else?

I'm referring to the growth rate of the full set of prefixes and the
capabilities of the control plane to support it.

Note that I've been saying this pretty consistently for two full years now.

|Do you mean
|"with a table of 100,000,000 entries, there will need to be at 
|a minimum
|some incremental update to existing deployed implementations" (to which
|my teenage daughter might say "duh!"), or do you mean "it won't be
|economically reasonable to build products unless we make some localized
|internal (within a router) change to a few algorithms?", or do you mean
|"it will be financially too expensive to build products that can handle
|some future table size", or do you mean "it will not be technically
|possible to build the routers", or do you mean something else? 

Once again, with feeling:

The growth rate of prefixes that are circulating within the DFZ exceeds the
rate of speed improvement in the underlying DRAM that we use to implement
the control plane.  This will undoubtedly result in architectural changes
and cost increases.  The concern then becomes that over the long term, as
this continues, the cost increases must be absorbed somewhere in the system,
and it is most likely to propagate to the end users, with a net detriment to
the growth of the network.

|Secondly, depending upon precisely what you mean by "faster than the
|technology can keep up", speaking as an individual, it is quite likely
|that I disagree. I hope that any disagreement is not automatically
|declared to be off-topic. 

You're more than welcome to disagree.  However, I hope that we can have a
more constructive discussion that simply reiterating the size of our
personal FIB.

Tony

---

From: "Tony Li" <[email-masked]>
Date: Sat, 3 Jan 2009 13:45:46 -0800

|> We already know that we're growing the table faster than 
|technology can keep
|> up.  That much is clear.
|
|Is that a falsifiable scientific theory or is it a consensus opinion
|without quantitative grounding?

Worse, that's an opinion, with some quantitative grounding, held by an
apparent minority.

|The well-grounded trend lines I posted suggest that the BGP table size
|at the edge of what we can build cost-effectively may be compounding
|at a rate of 40% per year while the actual size of the BGP table may
|only be compounding at around 25% per year. If true, one consequence
|should be that the minimum cost of an edge device capable of
|participating in the BGP backbone has fallen significantly. This
|appears to be the case; a $1400 Cisco 2811 is now capable of
|interacting with the full BGP table while 15 years ago you had to
|spend $60k on a Cisco 7000 to seriously think about a full BGP table.

More than something of a red herring, given that they are violently
different generations of hardware, with very different levels of hardware
optimization.  The 7000, for example, was built before the networking
industry was on to the ASIC bandwagon.  Since we've transitioned from
technology laggards to being near cutting edge, there have been some one
time advances.

|So no, the truth of the claim that we're growing the table faster than
|technology can keep up is not at all clear.

Fair enough.  

So, why do we keep debating this?  We've been over this to death.  The horse
is dead already.  For folks that feel that there's no problem, why not leave
the rest of us in peace to do what we feel we need to do?

Tony

---

From: "William Herrin" <[email-masked]>
Date: Sat, 3 Jan 2009 20:20:44 -0500

> technology laggards to being near cutting edge, there have been some one

The history of the computing industry is one of regularly encountered
one-time advances. Moore's Law isn't built on the back of purely
evolutionary improvement.

That today's router hardware is radically different than that of 15
years ago doesn't refute the sustainability of the growth trend; it
demonstrates it.

> the rest of us in peace to do what we feel we need to do?

I'm debating it because I'm tired of watching us bounce off the
pinball bumpers. Resolving this issue impacts our work here in three
very important ways:

1. It affects whether as a group we reject strategy F, and if we
don't, the language with which we present it.

2. Engineer's maxim: if it ain't broke, don't fix it. If failure does
not loom on BGP's horizon, there is little justification for moving
any disruptive strategy forward from research to engineering. The
desirable output from our efforts is *very* different depending on
whether or not BGP is failing.

3. RIRs make painful, real-world-hurtful decisions based on the
credence they lend the warning of BGP's impending failure. If the end
of the world is no longer nigh, we have a moral responsibility to let
them know.

#3 hits me particularly hard at work where an $800k continuing
operations component of a multi-million dollar project can't justify a
/24 from ARIN for multihoming.

However frightful the monsters under the bed may be, there is
insufficient proof of their existence to justify giving your toddler a
shotgun.

Regards,
Bill Herrin

---

From: Robin Whittle <[email-masked]>
Date: Sun, 04 Jan 2009 13:20:59 +1100

Short version:    The current DFZ growth rates are not our target.
                  In the long term - including ideally the next
                  6 to 10 years - we need to alter the architecture
                  so it can cope well with hundreds of millions of
                  multihomed end-user networks.

I eyeballed the graph:

  http://bgp.potaroo.net

and estimated 16 to 17% per annum growth in the last 3 or 4 years.

I expect this rate of growth will increase in the next few years as
IPv4 space is subdivided more intensively once the last of the fresh
pastures at the edge of town are engulfed.

I think there is little point in discussing whether DRAM speeds, CPU
speeds etc. can cope with the current rate of growth in the DFZ in
the long term.

In the long term, we need to enhance the architecture of the Net to
cope happily with a much larger growth in the numbers of end-user
networks which have multihoming, TE and portability.

I suggest we should be happy with a system which is technically and
commercially agreeable with numbers of such end-user networks like:

  2016      1M
  2020     16M
  2024    256M

This is 100% PA growth.  Each end-user network will probably have
multiple prefixes too, for load sharing over multiple links and for
their multiple sites - so the total number of portable, multihomable,
prefixes needed will be 2 to 10 times this.

Ideally, the scalable routing solution should be technically and
commercially capable of scaling within a few years of introduction to
a hundred million or so separate prefixes (or non-binary length
spans) of end-user network address space.  However, I guess it would
take a few years for such levels of demand to eventuate (not counting
the immense demand for stable IP addresses for mobile devices).

There's no way that CPUs and RAM, for either the RIB or the FIB, can
cope with this sort of growth, as long as the prefixes are provided
by the current BGP-based techniques.

Even if we could somehow build and afford the routers, this would be
an inelegant, unstable and probably unmanageable number of prefixes
to expect the interdomain routing system to cope with.

The current system is unacceptable since it doesn't fit the needs of
many end-user networks.  The disparity between the unadorned BGP
system's capabilities and the needs of end-user networks will get
worse with every passing year.

I don't think there is any brick wall where the interdomain routing
system as we run it today will stop working.  I am sure DFZ routers
can be built to handle 20% PA growth in DFZ prefixes.  (Despite what
I wrote about DRAM latency, routers could be built with faster
memories, including static RAM and/or with multiple CPU-RAM systems
sharing the RIB and FIB loads.)

Our success will be measured in how soon we can devise a better way
of providing hundreds of millions of end-user networks with
multihoming, TE and portability.

IPv4 address space will become harder and harder to get, since every
customer needs one such address.  A good core-edge separation scheme
could improve on this considerably, by enabling improved address
utilization.  It could do this by enabling many end-user networks to
multihome etc. with as small a number of addresses as they really
need - not in multiples of 256, 512 etc.

Another measure of our success will be how well the scalable routing
solution enables more efficient IPv4 address utilization.  That will
provide more time for figuring out an alternative to IPv4.

Ideally, a well designed solution to the routing scaling problem
could help with the transition to whatever lies beyond IPv4.

Ideally, it will help with mobility too.

  - Robin

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 04 Jan 2009 17:28:25 +1300

On 2009-01-04 14:20, William Herrin wrote:
...
> of the world is no longer nigh, we have a moral responsibility to let

The warning is a bit more subtle than that. It's more like (and I write
with no authority whatever):

If the DFZ growth curve reverts to the pattern it showed in the first
months of 1994, or in late 2000, or sustains the pattern it seems to have
shown since about mid-2006, it's liable to oustrip the increase in
performance that's possible by upgrading core routers. To prevent this,
continued damping of the growth in announced prefixes is essential.
RIR policy is the only damping mechanism proven to work.

    Brian

---

From: "Tony Li" <[email-masked]>
Date: Sat, 3 Jan 2009 21:48:21 -0800

Bill,

|The history of the computing industry is one of regularly encountered
|one-time advances. Moore's Law isn't built on the back of purely
|evolutionary improvement.

True, but one time advances from the networking industry catching up to the
rest of computing are all expended.  It now tracks it quite well, albeit
with some lag due to the ASIC houses trailing the process houses.  You do
NOT want router vendors going down the path of building their own fabs and
exploring process technology.  That's on the order of a $1B investment every
year.  ;-(

|I'm debating it because I'm tired of watching us bounce off the
|pinball bumpers. Resolving this issue impacts our work here in three
|very important ways:
|
|1. It affects whether as a group we reject strategy F, and if we
|don't, the language with which we present it.

That's easy.  

<co-chair>
As the RRG is chartered to deliver a routing architecture, strategy F (no
changes to the routing architecture) is out-of-scope.
</co-chair>

|2. Engineer's maxim: if it ain't broke, don't fix it. If failure does
|not loom on BGP's horizon, there is little justification for moving
|any disruptive strategy forward from research to engineering. The
|desirable output from our efforts is *very* different depending on
|whether or not BGP is failing.

Again, it's not BGP, it's the number of prefixes.  Any other protocol would
have to deal with the implications of a DFZ with a full set of prefixes as
well.  Thus, it's the architecture that must change.  And, once again, if
you don't believe that it needs to be fixed, I refer you to the problem
statement.

|However frightful the monsters under the bed may be, there is
|insufficient proof of their existence to justify giving your toddler a
|shotgun.

Hopefully there are better solutions than that regardless of motivation.  My
toddler has horrible aim.  ;-)

Tony

---

From: Danny McPherson <[email-masked]>
Date: Sun, 4 Jan 2009 14:41:41 -0700

> you don't believe that it needs to be fixed, I refer you to the  

Yep, and again, not strictly the number of prefixes,
but the number of paths, especially when one focuses on
RIB size and churn and implications on FIB updates and
an eye on some future security - versus sole focus on
"DFZ FIB size".

Many networks today have FIB sizes < 500k, but BGP
table sizes on the order of 5M or more.  As the denseness
of inter-domain interconnection continues to increases,
the number of paths increases (more steeply than the
DFZ table size).  A multi-homed AS with a PI prefix
might mean 100 *unique* paths or more in a network only
3 AS hops away.  Each change in reachability for that
prefix means all those paths have to be withdraw.

Ohh, and in an ideal world, those 100 viable paths would
have to be authorized in some manner.  Today this would
be done with static prefix lists at each interconnect,
although the current number of paths and absence of a
formally verifiable source for who's authorized (e.g.,
SIDR) to advertise (much less "transit" a given prefix)
what prefixes means there is essentially no inter-domain
route advertisement security in today's Internet.

More path attributes also means less efficiency in things
like update packing that allow some scaling of today's
routing system.

It's not just DFZ size that's the concern here.

-danny

---

From: "William Herrin" <[email-masked]>
Date: Sun, 4 Jan 2009 21:02:25 -0500

> rest of computing are all expended.  It now tracks it quite well, albeit

So routing improvement now "tracks quite well" with general computing
improvement which is predicted by Moore's Law to compound at 40%
annually, just like I claimed for routers?

> well.  Thus, it's the architecture that must change.

Please take it as a given that "BGP failing" is in fact shorthand
referring to the failure of the routing architecture in which every
multihomed system requires a slot in the routing tables of every core
router with all the attendant data transmission and computation. Et
cetera.

>  And, once again, if

I've long been convinced that BGP's scaling problem -should- be addressed.

Whether it -needs- to be addressed is an entirely different issue
which has a major practical impact on the intermediate research goals
and on the appropriate research posture that focuses our efforts.

Stance matters.

Proposals based on the presumption that BGP -needs- to be replaced
will far overreach proposals based on the proposition that we should
steadily nudge the architecture towards something more desirable.
Without a clear case that BGP -needs- to be replaced, that overreach
will almost certainly render the proposals into operational
non-starters.

As we overreach, we'll tend not to closely examine the more moderate
proposals for evolving in to an improved architecture that do not
immediately form into a comprehensive solution.

Tony, you've long complained that we're jumping straight to concrete
action proposals instead of doing research, but the research stance
based on the proposition that BGP will fail compels us to focus on
comprehensive solutions, for which concrete action proposals are
required.

> toddler has horrible aim.  ;-)

When we asked the RIRs obstruct PI prefixes for multihomed IPv4 and
IPv6 users, we upgraded them from cap guns to bullets. It's a very
*harmful* practice which should be discouraged absent testable and
quantitatively substantiated growth predictions projecting routing
table growth which outstrips our ability to build hardware. The IPv6
side is especially harmful, given the looming free pool crisis whose
projections are very well grounded. As a group, we should withdraw
that recommendation and discourage obstructing PI space for
multihomers.

Regards,
Bill Herrin

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 05 Jan 2009 16:34:00 +1300

On 2009-01-05 15:02, William Herrin wrote:
...
> IPv6 users, we upgraded them from cap guns to bullets.

That was in early 1994, I believe. If you were looking at the DFZ
growth curve back then, the bullets seemed inadequate, but they worked
for a while.

> It's a very
> projections are very well grounded. 

As I've always understood things, the challenge is to find the best
possible distribution of the harm; if the RIRs hadn't succeeded in
damping prefix growth, who knows when things would have ground to a
painful halt? My conclusion is exactly the opposite of yours: absent
substantiated growth predictions, the priority must continue to be
precautionary damping of the growth, as a matter of public importance.

> As a group, we should withdraw

Didn't you forget to add "as soon as there is a proven way to
ensure that the resulting growth in prefixes can be handled by the
routing system"?

(Not that the RRG can withdraw CIDR, anyway.)

    Brian

---

From: "Tony Li" <[email-masked]>
Date: Sun, 4 Jan 2009 21:50:08 -0800

|So routing improvement now "tracks quite well" with general computing
|improvement which is predicted by Moore's Law to compound at 40%
|annually, just like I claimed for routers?

Correct, but please again see my presentation.  Moore's law only applies to
transistor density (and it's not 40%), which doesn't necessarily help us.

|Please take it as a given that "BGP failing" is in fact shorthand
|referring to the failure of the routing architecture in which every
|multihomed system requires a slot in the routing tables of every core
|router with all the attendant data transmission and computation. Et
|cetera.

Please fix your misleading shorthand.

|Tony, you've long complained that we're jumping straight to concrete
|action proposals instead of doing research, but the research stance
|based on the proposition that BGP will fail compels us to focus on
|comprehensive solutions, for which concrete action proposals are
|required.

What I've tried to do is to encourage architectural discussion PRIOR to
going to the concrete.  There are lots of big upper layer issues (exactly at
the level of your strategies) that need to be dealt with before getting into
the bits and the bytes.

Tony

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 5 Jan 2009 06:13:22 -0800

Robin,

> networks which have multihoming, TE and portability.

Some folks think that the (large) majority of users will be mobile
devices. If that is indeed the case, then one should ask whether
the current solution for supporting host mobility (Mobile IP) is
viewed as adequate for this scenario. As if not, then enhancing the
architecture without taking into account the need to provide an
adequate support for a (very) large number of mobile hosts would
be rather myopic.
 
Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 5 Jan 2009 06:18:01 -0800

Robin,

> many end-user networks.  

The current system is *acceptable* to a very large number of users - for
number of users for whom the current system is acceptable just count
the number of users in the Internet.

Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 5 Jan 2009 07:09:49 -0800

Tony,

> the growth of the network.

Few points:

 - The above assumes that DRAM will continue to be used to implement control 
   plane memory. As had been discussed before on this list, DRAM is not the 
   only option  - another alternative is RLDRAM. 

 - Even if one assumes that the growth rate of prefixes exceeds the
   rate of speed improvements in the underlying DRAM that is used to
   implement control plane, and that, in turn, would result in cost
   increases, let's not forget that deploying new routing architecture
   may have its own cost, thus resulting in cost increases. Moreover,
   these increases may be higher relative to what would result from
   the growth rate of prefixes exceeding the rate of speed improvements
   in the underlying DRAM.

   These increases, just like the increases you mentioned above,
   would need to be absorbed by the system, and "most likely to
   propagate to the end users, with a net detriment to the growth
   of the network".  Moreover, these increases would need to be
   absorbed by the system in the short/medium term, well before we
   will reach "the promise" of the cost reduction in the *long
   term*. Would the industry be willing to buy into the *promise* of
   "a paradise" in the *long* term at the price of fairly certain
   cost increases in the *short/medium* term ? 

   All of the above means that a "promise" of cost reduction in the
   *long* term may not be sufficient - the new routing architecture
   needs to deliver in the *short/medium* term sufficient *tangible*
   benefits to justify its *short/medium* term cost.

 - One may argue that the real problem is not in the cost increases,
   but in the way how these increases are absorbed by the system.
   Namely, the real problem is that those who do not benefit
   are still required to pay for covering the cost imposed by those
   who do benefit. That applies both to the current system, as well 
   as to any proposal for a new routing architecture.

Yakov.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 05 Jan 2009 17:47:56 +0100

>     only option  - another alternative is RLDRAM.

So in other words, Tony should have said "architectural changes OR cost 
increases".  My recollection was that when the workshop happened, one of 
the key points was that we were not going to drop dead tomorrow, but 
simply ride the bleeding edge into oblivion as costs go up and up.  Now 
I know nothing about RLDRAM, but I'll hazard a guess that it is at least 
for now more expensive than DRAM.

>     the growth rate of prefixes exceeding the rate of speed improvements

Yes, the intersection of engineering and architecture is when you 
actually have to make design choices that require use of a crystal ball.

>     of the network".

I'm not convinced the two statements follow.  If the cost is a change in 
failure mode behavior, with associated mitigation techniques, perhaps 
there's a considerable net gain?  Early results with LISP show precisely 
this.  And we've discussed this before, so why are we looping?  If this 
is really an RG, let's not do that.

>    Moreover, these increases would need to be
To restate, vague generalities are not helpful.

>     as to any proposal for a new routing architecture.

Absolutely proven with BGP.  Do we have enough of an understanding of 
all proposals to make such sweeping statements?  Perhaps you could take 
us through that work and in particular specific value chain shifts and 
failures?

Eliot

---

From: Jason Schiller <[email-masked]>
Date: Mon, 05 Jan 2009 15:00:02 -0500 (EST)

>    as to any proposal for a new routing architecture.

It is important to point out that in the past the need driving the cost
increases to scale routers was either amount of bandwidth (as customers
wish to pass more traffic) or port density (as more customers desire to
connect).  Both of these come with an increase in revenue for the
provider, and as such (if the economics make sense) can justify the cost
of the upgrades.

We are currently faced with a need to upgrade routers due to the amount of
the routing state (and potentially the frequency with which it
changes).  These conditions are not necessarily a reflection of
increasing customer demand or increasing revue, and in fact a large
portion of this problem is originated by non-customers.

<insert Geoff Huston discussion on problems with a routing economy>

Charging non-customers per prefix to install routes in my routing table
seems like it would take the Network Neutrality problem to a whole new
level.

Finally WRT absorbing the cost within the system you will need to
consider:

1. What pricing model do vendors use for routers such that a moderately
low end CPE that can carry multiple views of full routes (think cisco
7200 for customer with a single T1 to each of four different
providers) remains affordable to small enterprises without making the cost
of high end routes grow logarithmically as they keep up with table growth.

2. Routers have to be of sufficient scale to provide large ISPs enough
time to complete certification, schedule maint windows and fully deploy
all of the new devices, and hopefully depreciate the assets for 5 years
before starting the cycle all over.

3. Are the added costs a one time event?  Can this even be roll into a
normal refresh cycle?  Or are the added costs recurring, such as upgrading
chassis every 3 years at a cost increase?

4. who should bear the burden of the extra cost?
Should ISPs that must be default free carry a larger burden?  

What about end-sites that are not required to be default free, but desire
it for better TE? 

Should ISPs that are default free with many routers carry a larger burden
than DFZ ISPs with few routers?  Consider that the number of routers in an
ISPs network does not necessarily impact the number of routes they source
into the DFZ.  

Should this cost just be passed along to an ISP's customers?  If so then
you reward the ISP that flood the DFZ with routes, and penalize the ISPs
who aggregate well.

__Jason

---

From: Robin Whittle <[email-masked]>
Date: Tue, 06 Jan 2009 12:09:17 +1100

Replying to    Yakov Rekhter:  Mobility for billions of users.

                               Whether the current system is
                               "acceptable".

                               RLDRAM (Reduced Latency Dynamic RAM)
                               is not much faster than DDR SDRAM.

               Eliot Lear:     RLDRAM costs.

Hi Yakov,

You wrote:

>> The current system is unacceptable since it doesn't fit the needs
>> of many end-user networks.
>
> The current system is *acceptable* to a very large number of users
> - for number of users for whom the current system is acceptable
> just count the number of users in the Internet.

OK - everyone who uses the Internet accepts it as it is, because
there is no alternative.

We are attempting to redesign the Net so that in the decades to come
the needs and desires of end-users will be better met.  I think one
important thing we need to provide is much easier access to
multihoming than is currently possible.  In that sense, I meant that
"the current system" is unacceptable - *we* shouldn't accept the
current difficulties with multihoming as something which should
remain in the decades to come.

This is a restatement of:

>> In the long term, we need to enhance the architecture of the Net
>> to cope happily with a much larger growth in the numbers of
>> end-user networks which have multihoming, TE and portability.

By "portability" I meant keeping the the network's address range when
selecting another ISP.  Renumbering can never be anywhere near as
inexpensive and risk-free as not having to renumber - in part because
network addresses appear in all sorts of places, including DNS files
and ACLs in other networks.

> devices.

I do, since I think there is a huge demand and that it will be
perfectly feasible to achieve this - with both IPv4 and IPv6.  IPv6
or at least something other than IPv4 is needed for billions of
cell-phone / media-players / PCs.

The desire, demand, need or whatever for physical mobility -
including with session continuity during all physical movements and
access network changes - is very strong indeed.  I think the demand
is worth hundreds of billions of dollars (to whoever satisfies it) -
just like in the cellphone industry.

> If that is indeed the case, then one should ask whether
> viewed as adequate for this scenario.

I don't think it is adequate for supporting global mobility with each
host retaining its IP address(es) year after year, while achieving
generally optimal path lengths for all packets.

> As if not, then enhancing the architecture without taking into
> account the need to provide an adequate support for a (very) large
> number of mobile hosts would be rather myopic.

I agree.  Steve Russert and I wrote:

  TTR Mobility Extensions for Core-Edge Separation Solutions to the
  Internet's Routing Scaling Problem  (2008-08-25)
  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

This imposes no architectural complications on the basic Ivip
core-edge separation scalable routing solution.  It extends it with a
type of ETRs called a Translating Tunnel Router.  Mobile hosts,
connecting by any means whatsoever - including Mobile IP - choose a
TTR physically and topologically nearby, such as 1000km or less.
They create one or more tunnels to the TTR, which acts as an ETR and
also sends outgoing packets to the rest of the Net.

There would be extra mapping changes in such a system, but the
mapping changes are only needed infrequently - such as when the
mobile node moves so much as to benefit from a closer TTR.  Mapping
changes are not required when the MN changes its access networks, as
long as they are all close nought to the current TTR.

In Ivip, each mapping change is paid for by the end-user - so the
mapping changes of mobile users will pay their way, and indeed help
make the overall Ivip system profitable for its operators.

Quoting Tony Li:

  http://www.irtf.org/pipermail/rrg/2009-January/000674.html

>    only option  - another alternative is RLDRAM. 

A 2006-02 article comparing Reduced Latency DRAM and Quad Data Rate
SRAM is:

http://www.networksystemsdesignline.com/shared/article/showArticle.jhtml?articleId=180200010

RLDRAM provides only marginal improvements.  There is still a long
cycle time due to the need to precharge the sense amplifiers.
Splitting the chip into different independent segments helps
somewhat, but the cycle time is still a limitation when reading or
writing to the same segment in quick succession.

QDR SRAM has no such problem - reads and writes can follow one after
the other every 3nsec or so, as I wrote:

  http://www.irtf.org/pipermail/rrg/2008-December/000632.html

Looking at the latest Micron parts

  http://www.micron.com/products/dram/rldram/partlist

There is a 575M bit RLDRAM II chip and less than 2ns "cycle times" -
but that is the interface cycle time.  Looking at the datasheet:

  http://download.micron.com/pdf/datasheets/rldram/MT49H16M36A.pdf

the cycle time between successive reads is 15ns.

This is only marginally less than for the latest PC-style DDR3 SDRAMs:

  http://www.micron.com/products/dram/ddr3/partlist.aspx

The fanciest of these is the 4G bit (twin chip) MT41J1G4.  The timing
information is found in the top link at the right from:

  http://www.micron.com/products/partdetail?part=MT41J512M8THU-15E

A 170 page data sheet for a memory chip . . .

I spent 15 minutes looking at this and couldn't easily find the cycle
time of interest.  The read latency (CL) figures on page 1 are 13ns
to 15ns.  This is the delay in producing the first read data.  The
total cycle time before another read or write could be performed
would be longer than this.  I guess 20 to 25 nsec.

RLDRAMs are less dense (a factor of 4 is evident in the above chips)
than DDR3 SDRAM.  This is partly a function of the massive demand for
the latter, which is likely to remain, since SDRAMs are highly
optimised for filling the on-chip caches of Pentium, Core, AMD etc.
mass market 32/64 bit CPUs.

I agree with the rest of your message.

However, I think our target shouldn't be keeping up with current
rates of growth in multihomed end-user networks, but in meeting the
real demand - which is much higher than the small subset who overcome
the currently very steep cost and administrative barriers to get
their own PI prefixes advertised in the DFZ.

Eliot wrote:

> Now I know nothing about RLDRAM, but I'll hazard a guess that it is
> at least for now more expensive than DRAM.

You can bet your sweet bippy on that - now and in the future.

As noted above, the latest Micron RLDRAM chips have 1/4 the capacity
of their DDR3 SDRAM contempories.

RLDRAM only provides marginal speed increases, but the chips have
fancier interfaces, since they allow all address bits to presented in
a single cycle, in contrast to the decades old DRAM tradition of
clocking the address bits in two cycles on half the number of pins.

Digikey has Micron 288M bit RLDRAM (MT49H8M36FM-25 TR in a 144 "pin"
Ball Grid Array package) for USD$52 in 1k quantities.

A Micron 1G bit DDR3 DRAM (MT41J256M4HX-187E:D TR in a 78 "pin" BGA
package) costs USD$9.40 in 2k quantities.

On this basis, the raw cost ratio between RLDRAM and DDR3 SDRAM is:

  (52 / 288) / (9.4 / 1024) = 19.6.

However, the larger size of the RLDRAM packages, and the fact that
four times as many packages are needed for a given amount of data,
means the real cost of using RLDRAM relative to DDR3 DRAM is higher
still.  Furthermore, RLDRAM would involve greater power supply costs,
heat dissipation, cooling costs etc.

Since RLDRAM provides only a marginally faster total cycle time
between reads or writes, I would say these costs are prohibitive.

A better option would be to go straight to QDR SRAM, which is
probably the same or somewhat higher price than RLDRAM, and has a
much shorter cycle time (3ns) between successive reads and/or writes.

 - Robin

---

From: "William Herrin" <[email-masked]>
Date: Mon, 5 Jan 2009 20:38:06 -0500

> On Mon, Jan 5, 2009 at 3:00 PM, Jason Schiller <[email-masked]> wrote:

Jason,

I'm picking a nit but...

A Cisco 7200 is overkill for a customer today running 4 BGP sessions
over 4 T1's. A 2811 is more than capable of the task. For anyone not
familiar with it, the 2811 is this generation's version of the 2500
series. In the described configuration it costs $2000-$3000 depending
on how carefully you shop.

Regards,
Bill Herrin

---

From: "William Herrin" <[email-masked]>
Date: Mon, 5 Jan 2009 20:40:29 -0500

On Mon, Jan 5, 2009 at 9:13 AM, Yakov Rekhter <[email-masked]> wrote:
> enhancing the

Ditto, assuming BGP's architecture is not collapsing.

Regards.
Bill Herrin

---

From: "Tony Li" <[email-masked]>
Date: Tue, 6 Jan 2009 11:36:11 -0800

Hi Yakov,

| - The above assumes that DRAM will continue to be used to 
|implement control 
|   plane memory. As had been discussed before on this list, 
|DRAM is not the 
|   only option  - another alternative is RLDRAM. 

I think that Robin did an excellent job of expanding on the tradeoffs
involved in this, and I agree with his conclusion.

I'll also point out that that's at best a one time benefit and that the
subsequent speed increases would seem likely to parallel the DRAM speed
curve.  This is likely since RLDRAM is still using the same basic DRAM cell
technology and improving on the electrical interface.  Thus, RLDRAM would
give us a one-time bump up the speed curve, but it would do nothing to
change the shape or slope of the curve.

Again, the most important thing here is the slope (or more generally the
shape) of the curve.  As long as the slope of the prefix curve exceeds the
slope of the technology curve, you will eventually and necessarily run into
a problem.

| - Even if one assumes that the growth rate of prefixes exceeds the
|   rate of speed improvements in the underlying DRAM that is used to
|   implement control plane, and that, in turn, would result in cost
|   increases, let's not forget that deploying new routing architecture
|   may have its own cost, thus resulting in cost increases. Moreover,
|   these increases may be higher relative to what would result from
|   the growth rate of prefixes exceeding the rate of speed improvements
|   in the underlying DRAM.

True, but deploying a new architecture will have a one-time cost impact.
Paid for, over and done.  It will be spread out over each system that needs
to be upgraded, but it is a fixed and finite cost.  

Paying for a more expensive control plane is likely to be an ongoing, ever
increasing cost.

|   Would the industry be willing to buy into the *promise* of
|   "a paradise" in the *long* term at the price of fairly certain
|   cost increases in the *short/medium* term ? 
|
|   All of the above means that a "promise" of cost reduction in the
|   *long* term may not be sufficient - the new routing architecture
|   needs to deliver in the *short/medium* term sufficient *tangible*
|   benefits to justify its *short/medium* term cost.

One might look it at it instead as avoiding the guarantee of long term
higher costs.

In any case, there are numerous problems that we do feel we need to overcome
with the new architecture.  Whether the deployment costs are sufficiently
justified will in part depend on what we produce, and we will not know that
until we have consensus on a workable architecture.

| - One may argue that the real problem is not in the cost increases,
|   but in the way how these increases are absorbed by the system.
|   Namely, the real problem is that those who do not benefit
|   are still required to pay for covering the cost imposed by those
|   who do benefit. That applies both to the current system, as well 
|   as to any proposal for a new routing architecture.

Agreed.  The fundamental problem is that the architecture is a ready-made
tragedy of the commons.

Tony

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 07 Jan 2009 10:03:29 +1300

Tony,

On 2009-01-07 08:36, Tony Li wrote:
...
> slope of the technology curve, you will eventually and necessarily run into

I know where to find data for the prefix curve, although it's very
nosiy data and there is no reliable model that allows us to fit the
curve and extrapolate. But where do I find useable data for the
technology curve? Marketing-style hockey sticks don't help.

    Brian

---

From: "Christopher Morrow" <[email-masked]>
Date: Tue, 6 Jan 2009 17:05:22 -0500

> nosiy data and there is no reliable model that allows us to fit the

and as always remember that internal routing table growth happens
outside of globally visible changes, and often (ATT's CBB as an
example, sprint/nextel IMS deployments as another) is completely
internal... unfortunately they are also not as predictable. I would
guess that internal tables at most providers (based on the preso's
from AMS/RAWS) are 30-50% larger than the global table. Some providers
(ATT/Sprint) are looking at vastly larger internal tables.

-Chris

---

From: "Tony Li" <[email-masked]>
Date: Tue, 6 Jan 2009 15:23:50 -0800

Hi Chris,

|and as always remember that internal routing table growth happens
|outside of globally visible changes, and often (ATT's CBB as an
|example, sprint/nextel IMS deployments as another) is completely
|internal... unfortunately they are also not as predictable. I would
|guess that internal tables at most providers (based on the preso's
|from AMS/RAWS) are 30-50% larger than the global table. Some providers
|(ATT/Sprint) are looking at vastly larger internal tables.

I know that the internal tables are always a pain, but since we have to deal
with the global issues, the internal, private growth (self-inflicted ;-) has
to necessarily be out of scope for RRG.

Tony

---

From: "Christopher Morrow" <[email-masked]>
Date: Tue, 6 Jan 2009 20:59:36 -0500

> to necessarily be out of scope for RRG.

Sure, my point was that we should keep in mind that the 280k we see
externally today is a shadow of the internal situation, and if the
'new architecture' can address both internal and external, or make the
external much less painful then lots of things get easier. (could get
easier)

-Chris

---

From: Danny McPherson <[email-masked]>
Date: Mon, 12 Jan 2009 08:07:53 -0700

> to necessarily be out of scope for RRG.

That's quite odd to me, considering by today's definitions
"internal [BGP] tables" are where the routing scalability and
stability are it's worst - today, and unquestionably, the first
place things will break IF/WHEN they do, as a result of an
inter-domain routing protocol architecture that forces either
full-mesh or hierarchies such as route reflection that themselves
introduce additional paths and state in the network (even with
implicit aggregation effects).

And they're not going to break because of a 100k unique
internal-only routes, they're going to break because of
an order or magnitude or more paths (and all of their overhead)
- paths introduced as a result of "global issues" and solutions
that focus on solely minimizing DFZ size, rather than looking
at where the problem is actually worst - today.

-danny
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: "Tony Li" <[email-masked]>
Date: Mon, 12 Jan 2009 09:34:34 -0800

Hi Danny, 

|> I know that the internal tables are always a pain, but since 
|we have  
|> to deal
|> with the global issues, the internal, private growth (self- 
|> inflicted ;-) has
|> to necessarily be out of scope for RRG.
|
|That's quite odd to me, considering by today's definitions
|"internal [BGP] tables" are where the routing scalability and
|stability are it's worst - today, and unquestionably, the first
|place things will break IF/WHEN they do, as a result of an
|inter-domain routing protocol architecture that forces either
|full-mesh or hierarchies such as route reflection that themselves
|introduce additional paths and state in the network (even with
|implicit aggregation effects).

As you point out, this is more to do with the specific protocol than it is
to do with the overall routing architecture.  As such, this is why it's
really much more of the domain of IDR than RRG.

BTW, AFAIK, route reflection _reduces_ the number of paths as compared to
full-mesh IBGP.  Am I missing something?

|And they're not going to break because of a 100k unique
|internal-only routes, they're going to break because of
|an order or magnitude or more paths (and all of their overhead)
|- paths introduced as a result of "global issues" and solutions
|that focus on solely minimizing DFZ size, rather than looking
|at where the problem is actually worst - today.

Paths that are introduced for the sake of traffic engineering are a well
understood and self-inflicted problem.  If those folks that introduced the
extra paths took equivalent interest in limiting their dispersion and
cooperated in filtering unnecessary long prefixes, this could readily be
addressed.  In short, this is not an issue with the fundamental
architecture, it's about how the architecture is being used.  Thus, this is
largely an operational issue.  

Further, if we did something to fundamentally remove the capability to
traffic engineer in the future, I strongly suspect that that would be an
architectural issue, most likely fatal.

Regards,
Tony

---

From: "Christopher Morrow" <[email-masked]>
Date: Mon, 12 Jan 2009 12:47:21 -0500

> really much more of the domain of IDR than RRG.

So, it's not that RRG doesn't acknowledge this problem, it's that the
problem ought to be fixed (to some extent at least) with the work RRG
is doing?

> understood and self-inflicted problem.  If those folks that introduced the

I don't think it's just 'traffic engineering' that is the problem
here, it's often other things like L3vpn/2547-vpn/mpls-vpn, 'internal
services' or 'service networks' (Peter's notes about phones on IP
roaming around the sprint core).

Surely TE prefixes are issues, but that's not the only source of
internal route growth... one large one is 'this IP thing, it's kinda
successful!' (convergence).

-Chris

(btw, I support the idea that RRG's work will 'fix this problem', my
original note was just to continue to raise awareness that the numbers
for route growth need to keep in mind that 285k prefixes globally is
often dwarfed by internal table sizes.)
_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: "Tony Li" <[email-masked]>
Date: Mon, 12 Jan 2009 10:29:34 -0800

Hi Chris,

|So, it's not that RRG doesn't acknowledge this problem, it's that the
|problem ought to be fixed (to some extent at least) with the work RRG
|is doing?

Yes, of course the problem exists, but again, it's more to be fixed within
IDR.  BGP is not an architecture; it's a protocol.  Protocol changes are in
scope for IDR and architecture changes are in scope for RRG.

|> Paths that are introduced for the sake of traffic 
|engineering are a well
|> understood and self-inflicted problem.  If those folks that 
|introduced the
|
|I don't think it's just 'traffic engineering' that is the problem
|here, it's often other things like L3vpn/2547-vpn/mpls-vpn, 'internal
|services' or 'service networks' (Peter's notes about phones on IP
|roaming around the sprint core).
|
|Surely TE prefixes are issues, but that's not the only source of
|internal route growth... one large one is 'this IP thing, it's kinda
|successful!' (convergence).

VPN prefixes are also wholly self-inflicted and have been well discussed as
a scalability issue for a decade or more.  No one is required to run VPNs,
so it's also a self-inflicted problem, just as if you decided to run DECnet.
;-)

Tony

---

From: "Christopher Morrow" <[email-masked]>
Date: Mon, 12 Jan 2009 14:47:37 -0500

> scope for IDR and architecture changes are in scope for RRG.

ugh, my wording was unclear, but I think we are agreeing. ok, all done :)

> so it's also a self-inflicted problem, just as if you decided to run DECnet.

hrm.. 'self-inflicted'... we could have a long discussion about why
this IP thing has been successful, and how, and where it may go in the
future, and why things like vpn-routes are perhaps important to
that... but, like above, I think we agree that the problem could be
fixed with an architecture shift and that's the point of RRG.

Apologies for the 7 extra messages about internal/external prefixes
and sizing.  back to William's paper!

-Chris

---

From: Danny McPherson <[email-masked]>
Date: Mon, 12 Jan 2009 14:01:24 -0700

> really much more of the domain of IDR than RRG.

Well, so long as the folks designing the next generation
understand the problem and mitigate the constraints.  I'm
not totally sure the former is the case, which has effects
on the latter.

> full-mesh IBGP.  Am I missing something?

Depends on the number of RRs, where the best path
is, network architecture, etc..  The growth curves
I've seen (which I hope to be able to share soon)
illustrate that the number of iBGP paths is growing
at anywhere from 6-14x the rate of the DFZ, and
the reason is because of external factors, protocol
design issues, and implementations.

> understood and self-inflicted problem.

Really?  So a multi-homed AS is TE?  So networks that interconnect
in 10 places result in 10 paths for each prefix and that's self-
inflicted?  What's the alternative to avoiding this self-inflicted
pain?

>  If those folks that introduced the
> addressed.

No, I'm not talking about just extraneous or covered prefixes,
that's only a tiny part of the problem I'm referring to.
Denseness of external interconnection, unique attributes,
BGP as defined today AND TE have effects on these, I'd say the
latter the least.

>  In short, this is not an issue with the fundamental

I don't agree with this.  Offer me an alternative for 10 or
more interconnections between the largest networks today?
Offer me an alternative for how those paths and the current
inter-domain routing protocol amplify (not minimize) the number
of paths.

> architectural issue, most likely fatal.

Again, TE is only a tiny part of the problem I'm referring
to here - fully dependent on ones definition of TE, of course.

-danny

---

From: Bruce Curtis <[email-masked]>
Date: Tue, 13 Jan 2009 09:56:09 -0600

One of the limits discussed in this thread was that the performance  
of DRAM will be limited by it's dependence on capacitors.

   While Floating Body Memory isn't yet mainstream and can't yet be  
counted on to solve future problems I was surprised that it wasn't at  
least mentioned.

   http://www.spectrum.ieee.org/oct08/6868

   http://en.wikipedia.org/wiki/Z-RAM

   http://en.wikipedia.org/wiki/TTRAM

   http://www.spectrum.ieee.org/jan07/4839

   http://www.innovativesilicon.com/technology_overview.php

   http://www.innovativesilicon.com/technology_benefits.php

   http://www.innovativesilicon.com/  (watch video overview)

   http://www.eetimes.com/showArticle.jhtml;jsessionid=BP5AO2PNJBYIEQSNDLOSKH0CJUNN2JVN?articleID=210604764

---
Bruce Curtis                         [email-masked]
Certified NetAnalyst II                701-231-8527
North Dakota State University