Subject: RE: Memory at PE

From: Andrew Walker <[email-masked]>
Date: Wed, 24 Apr 2002 06:18:23 -0700

Sachin,

You have asked a very important question as it gets to the
heart of the scalability of a PE deployment. In a traditional router
architecture (e.g. Cisco, Juniper, Unisphere), although FIBs are 
distributed out to individual line cards, the RIB is kept on a central
module. There is only one RIB and therefore all updates whether for the 
Local RIB, or for each VRF must go through this RIB. 
Obviously the RIB is divided into pieces so as to keep the 
VRFs separate, but the whole thing is bounded by the 
amount of memory and CPU cycles available to the routing plane.
Prior to BGP/MPLS VPNs, update performance of the RIB was 
not a large concern, so the fact that it was still centralized did not
matter.

However BGP/MPLS VPNs place a new burden on traditional routers. 
VRFs are by definition specific to a particular subscriber network and 
must only contain routes specific to that network; traditional routers 
partition their RIBs and FIBs in order to maintain this separation. This 
can become a problem for traditional routers since the RIB is maintained 
in a central module and serviced by a single processor. These centralized 
resources (memory and CPU) become a bottleneck as the number 
of subscribers increases, especially when the PE-CE connections use chatty
routing protocols such as RIP or computationally intensive routing protocols

such as OSPF. For example, when using OSPF as the 
CE-PE routing protocol, a Cisco router can only sustain approx 20 to 30
instances per device. With a distributed architecture, like the 
CoSine IPSX platform, you can have more than 1500 such instances
per blade.

In a truly distributed device architecture, the RIBs are distributed
with the VRFs, and new resources (CPUs and Memory) can be 
added as needed.  Thus you avoid the limitations imposed by
traditional routing architectures.

The base architecture of a PE device is absolutely critical in 
determining how many BGP/MPLS VPNs the device can handle. 
Trying to retrofit an existing architecture for a new network design
can lead to limitations that purposely designed hardware doesn't suffer
from. 
--------

At 12:49 PM 4/23/2002 -0400, Liu, Chia J (Charlie), ALCNS wrote:
>Sachin,
>
>There are two aspects:
>*       Memory Usage in GRP/RSP:   In the lab, we saw data of ~935 bytes
per VPN prefix in VRF, compared to 500-600 bytes per internet prefix in
global routing table.    I heard there is additional 60KB-70KB overhead per
VRF.
>*       Memory Usage in Line Card:   We are particularly concerned about
the VRF memory overhead in PSA/TLU of E2 16xOC-3 in GSR.    It looks like
the number is different in different IOS releases.    I am interested in
knowing if anyone has number on this.   Thanx.
>
>C.J. (Charlie) Liu
>
>
>> -----Original Message-----
>> From: Sachin Kalra [SMTP:[email-masked]]
>> Sent: Tuesday, April 23, 2002 11:51 AM
>> To:   [email-masked]
>> Subject:      Memory at PE
>> 
>> Dear Group:
>> 
>> I would appreciate if I can get answer to the following question
regarding 
>> BGP/MPLS VPNs (RFC2547bis)
>> 
>> I understand that PE router maintains a separate VRF table for each VPN 
>> site connected to it. I wanted to know if a PE also maintain separate
RIBs 
>> for each VPN site, apart from its main Local RIB? Or, does it maintain
only 
>> one single RIB?
>> 
>> Actually, I was looking from the perspective of amount of memory required

>> at PE, if it has to maintain many VRFs and many RIBs.
>> 
>> Thanks for your response.
>> Sachin Kalra 
############################################################################
########################## This email communication may contain CONFIDENTIAL
INFORMATION and is intended only for the use of the intended recipients
identified above.  If you are not the intended recipient of this
communication, you must not use, disclose, distribute, copy or print this
email. If you have received this communication in error, please immediately
notify the sender by reply email, delete the communication and destroy all
copies.
############################################################################
##########################

---

From: "Jim Guichard" <[email-masked]>
Date: Wed, 24 Apr 2002 10:19:34 -0400

RE: Memory at PEAndrew,

I do not normally respond to these marketing type emails but I think it is
important that we try and stay within the realms of reality. There are a
whole bunch of things that contribute to scalability and to suggest that
central CPU/memory is the bottleneck, or only consideration for scaling, is
completely misleading. Jim
  -----Original Message-----
  From: [email-masked] [mailto:[email-masked]]On Behalf Of Andrew
Walker
  Sent: Wednesday, April 24, 2002 9:18 AM
  To: Sachin Kalra; Liu, Chia J (Charlie), ALCNS
  Cc: [email-masked]
  Subject: RE: Memory at PE

  Sachin,

  routing protocols such as RIP or computationally intensive routing
protocols

regarding
RIBs
only

---

From: "Miguel Angel Chana" <[email-masked]>
Date: Wed, 24 Apr 2002 16:49:31 +0200

RE: Memory at PEHello, Jim:

I cannot be more in agreement with you. This kind of messages in this list
are not
very useful, and I think we should not discuss particular implementations.
For me
this should not be a place for manufacturers to discuss who does it better.

Having said that, I can't resist to comment that every architecture has its
own
share of issues.

In a virtual router architecture, if there are N instances of routing, every
one receives more or less 1/N CPU process time and 1/N memory. Both
resources
are finite, unfortunately.

Regards

  -----Original Message-----
  From: [email-masked] [mailto:[email-masked]]On Behalf Of Jim
Guichard
  Sent: Wednesday, April 24, 2002 4:20 PM
  To: Andrew Walker; Sachin Kalra; Liu, Chia J (Charlie), ALCNS
  Cc: [email-masked]
  Subject: RE: Memory at PE

  Andrew,

    -----Original Message-----

    Sachin,

    in a central module and serviced by a single processor. These
centralized
    of subscribers increases, especially when the PE-CE connections use
chatty

    At 12:49 PM 4/23/2002 -0400, Liu, Chia J (Charlie), ALCNS wrote:
    >Sachin,
    >
    >There are two aspects:
    >*       Memory Usage in GRP/RSP:   In the lab, we saw data of ~935
bytes per VPN prefix in VRF, compared to 500-600 bytes per internet prefix
in global routing table.    I heard there is additional 60KB-70KB overhead
per VRF.

    >*       Memory Usage in Line Card:   We are particularly concerned
about the VRF memory overhead in PSA/TLU of E2 16xOC-3 in GSR.    It looks
like the number is different in different IOS releases.    I am interested
in knowing if anyone has number on this.   Thanx.

regarding
RIBs
    >> for each VPN site, apart from its main Local RIB? Or, does it
maintain only