Subject: Scalable and manageable network management and operation of MPLS/ VPN

From: "Chen, Weijing" <[email-masked]>
Date: Fri, 22 Nov 2002 12:07:08 -0600

Yakov,

Sorry for late response. I didn't receive your reply for somewhat reason.
Yetik pointed this message to me and I found it from mailing list archive.

>>Could you please elaborate on what exactly you are concerned with respect
to "scaleable manageability of MPLS/VPN"?

In a nutshell, RFC 2547 doesn't scale because it breaks the rule that all we
need to do to manage a subscriber is manage the attributes on the port
serving the subscriber.  We have to dedicate resources to the subscriber
that go deeper into the network than just their port.  Establishing, keeping
track of, and troubleshooting these resources on a per-subscriber basis will
be difficult.

Specifically, the resources we have to dedicate and manage are:

1.	Virtual Routing Functions.  Nearly one for each CE/PE interface.
Each one of these, from a management standpoint, is a router.  Right now,
large IP carriers manage hundreds to possibly a few thousand routers.  With
IP VPNs, we'll be managing tens of thousand to possible hundreds of
thousands of routers (VRF), a good 2 orders of magnitude jump.
2.	Connections between VRFs.  Each VRF in a VPN has to have a
connection to every other VRF in the VPN.  The more VRFs you have (100,000),
the more connections you have between them.
3.	Connections between PEs.  Each PE in a VPN has to have a connection
to every other PE in the VPN.  This connection maybe shared by multiple VRFs
in the same PE pairs.  However, tracking of what connection to be shared or
establishing new one if not shared is another daunting task.

Each single item above is translated into $$, which including recurring OPEX
for operation and OSS system maintenance, and nonrecurring CAPEX for OSS
system.

>>Also, when you said "we operation group in service provider", do you mean
a specific service provider?

Any service provider with hundreds of POPs (or COs), thousands of PEs
(routers or switches), millions of subscribers (number of sites), such as us

Regards,

--

Weijing Chen

SBC Technology Resources

9505 Arboretum Blvd.

Austin, TX 78759

512 372 5710

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 22 Nov 2002 19:23:57 +0100

Weijing,

Your below email clearly indicates that you are mixing VR based L3VPNs
with the L3VPNs described in 2547bis architecture as non of your points
apply to the latter one. I must admit that they are very true for the
first type of L3VPNs though :-).

Thx,
R.

> Yakov,
> 
> 
> 
>    Sorry for late response. I didn't receive your reply for somewhat reason. Yetik pointed this message to me and I found it from mailing list 
> archive.
> 
> 
> 
>    >>Could you please elaborate on what exactly you are concerned with respect to "scaleable manageability of MPLS/VPN"?
> 
> 
> 
>      In a nutshell, RFC 2547 doesn't scale because it breaks the rule that all we need to do to manage a subscriber is manage the 
> attributes on the port serving the subscriber.  We have to dedicate resources to the subscriber that go deeper into the network than just 
> their port.  Establishing, keeping track of, and troubleshooting these resources on a per-subscriber basis will be difficult.
> 
> 
> 
>    #Virtual Routing Functions.  Nearly one for each CE/PE interface.  Each one of these, from a management standpoint, is a 
>      router.  Right now, large IP carriers manage hundreds to possibly a few thousand routers.  With IP VPNs, we'll be managing tens of 
>      thousand to possible hundreds of thousands of routers (VRF), a good 2 orders of magnitude jump.
>    #Connections between VRFs.  Each VRF in a VPN has to have a connection to every other VRF in the VPN.  The more VRFs 
>      you have (100,000), the more connections you have between them.
>    #Connections between PEs.  Each PE in a VPN has to have a connection to every other PE in the VPN.  This connection 
>      maybe shared by multiple VRFs in the same PE pairs.  However, tracking of what connection to be shared or establishing new one if 
>      not shared is another daunting task.
> 
> 
> 
>      Each single item above is translated into $$, which including recurring OPEX for operation and OSS system 
> maintenance, and nonrecurring CAPEX for OSS system.
> 
> 
> 
>    >>Also, when you said "we operation group in service provider", do you mean a specific service provider?
> 
>    Any service provider with hundreds of POPs (or COs), thousands of PEs (routers or switches), millions of 
> subscribers (number of sites), such as us
> 
> 
> 
>    Regards,
> 
> 
> 
> 
> 
>    --
> 
>    Weijing Chen
> 
>    SBC Technology Resources
> 
>    9505 Arboretum Blvd.
> 
>    Austin, TX 78759
> 
>    512 372 5710
> 
> 
> 
> 
> 
>

---

From: Yakov Rekhter <[email-masked]>
Date: Fri, 22 Nov 2002 11:43:39 -0800

Weijing,

> track of, and troubleshooting these resources on a per-subscriber basis will

In a nutshell, quite a few service providers deployed 2547 for
quite some time, and thus have *practical* experience with it.  At
MPLS-2001 conference there was a presentation by Chris Chase from
AT&T (one of the service providers that deployed 2547 for quite
some time), titled "Scaling BGP/MPLS VPN Service And Comparison to
Traditional Layer 2 VPN Services". The presentation has a fairly
detailed analysis of the scalability aspect of 2547. I am not going
to repeat it here, but just to quote from the summary slide of that
presentation:

   Certainly scalable and solvable

> thousands of routers (VRF), a good 2 orders of magnitude jump.

With provider-provisioned Layer 3 VPNs a provider *by definition
of the service* has to participate in each VPN customer's routing.
This is irrespective of whether one uses 2547, Virtual Routers, or
(provider-provisioned) IPSec between CEs. So, the complexity of
Layer 3 VPN service has nothing to do with MPLS, but has to do with
the service itself.

If a service provider does not want to participate in VPN customers'
routing, the provider could either (a) offer no VPN services at
all, or (b) offer Layer 2 VPN services (e.g., FR, ATM, VPLS, etc...).
All of the Layer 2 VPN services I mentioned could be offered with
MPLS. So, if you think that Layer 2 VPN services do not have the
"scaleable manageability" problem, then clearly MPLS/VPNs (used to
offer Layer 2 VPN services) would not have the "scaleable manageability"
problem either.

In general, when comparing scalability of Layer 2 VPN service vs
Layer 3 VPN service, it is important to look at scalability of
*total system*, and not just scalability in the service provider.
There is a pretty good presentation from Bruce Davie (Cisco) at
the Spring 2002 MPLSCon Conference on this topic.

> the more connections you have between them.

Your assertion that "each VRF in a VPN has to have a connection to
every other VRF in the VPN" is incorrect. It shows that your
understanding of 2547 is inadequate. Please read
draft-ietf-ppvpn-rfc2547bis-03.txt.

> establishing new one if not shared is another daunting task.

Your assertion that "each PE in a VPN has to have a connection
to every other PE in the VPN" is incorrect. If you would read
2547 spec (draft-ietf-ppvpn-rfc2547bis-03.txt), you would find
in Section 4.3.3:

   Rather than having a complete IBGP mesh among the PEs, it is
   advantageous to make use of BGP Route Reflectors [BGP-RR] to improve
   scalability.  All the usual techniques for using route reflectors to
   improve scalability, e.g., route reflector hierarchies, are
   available.

> for operation and OSS system maintenance, and nonrecurring CAPEX for OSS

May I suggest you to get more familiar with 2547 before making any more
assertions about it. 

Yakov.

---

From: Thomas Nadeau <[email-masked]>
Date: Fri, 22 Nov 2002 14:49:34 -0500

At 12:07 PM 11/22/2002 -0600, Chen, Weijing wrote:

> >>Could you please elaborate on what exactly you are concerned with 
> respect to "scaleable manageability of MPLS/VPN"?
>
>
>
>In a nutshell, RFC 2547 doesn't scale because it breaks the rule that all 
>we need to do to manage a subscriber is manage the attributes on the port 
>that go deeper into the network than just their port.  Establishing, 
>keeping track of, and troubleshooting these resources on a per-subscriber 
>basis will be difficult.

         If you look at network management this way, then of course, any 
VPN technology you
look at will not be scalable.   However, the success of 2547 is indeed that 
a well designed
management system can managed new customers by managing basically the VRF 
attributes.
The typical addition of one (or hundreds) of customers should not affect 
the layout of your internal
network or even the PE where those customers are attached if you did things 
correctly
from the start.    For example, many providers I know will size how many 
customers a
particular PE can handle in their lab.  Once they know this, they just 
provision
customers in a single, easy way. Once they get near the limit, they order 
another
PE. This also plays into the larger design of the core network of course, 
but I think
that if your network is designed to slimly that it needs to be 
re-provisioned whenever
any customer is added, that you should think about your design again.

>    * Virtual Routing Functions.  Nearly one for each CE/PE 
> interface.  Each one of these, from a management standpoint, is a 
> router.  Right now, large IP carriers manage hundreds to possibly a few 
> thousand routers.  With IP VPNs, we'll be managing tens of thousand to 
> possible hundreds of thousands of routers (VRF), a good 2 orders of 
> magnitude jump.
         Perhaps you can explain to me why you think that the addition of 
one customer is going to
affect all of the nodes in your network?  I do not know of any deployments 
where providers
re-provision their entire network based on the addition or re-provision of 
a 2547 customer.
>    * Connections between VRFs.  Each VRF in a VPN has to have a 
> connection to every other VRF in the VPN.  The more VRFs you have 
> (100,000), the more connections you have between them.
         You are thinking about this as if MPLS is a connection-oriented 
network, which it
is not. The exception is if you are using RSVP-TE tunnels between 
PEs.  However,
a typical 2547 network uses LDP to establish sessions and distribute
labels between PEs, not each VRF.  In essence, you get paths between PEs
over which you switch the VPN labels. I think that you are thinking of things
as if there are actual connection-oriented paths between all VRFs in a VPN.
Even if you use RSVP-TE tunnels between PEs, you can
support many hundreds or thousands of VPNs between those PEs without needing
a TE tunnel per VRF inter-connection. Thus only a small number of tunnels 
exist
between PEs (as opposed to specific tunnels for each VPN or VRF-pair).
>    * Connections between PEs.  Each PE in a VPN has to have a connection 
> to every other PE in the VPN.  This connection maybe shared by multiple 
> VRFs in the same PE pairs.
         Yes, but why is this a problem?
>    * However, tracking of what connection to be shared or establishing 
> new one if not shared is another daunting task.
         Just look at the loopbacks at one VRF and that gives you pointers 
to the other PEs.
If you are interested in figuring out which physical links are shared by 
which VRFs on
a particular PE, then just look at the LFIB based on the label used by the 
loopback.

>Each single item above is translated into $$, which including recurring 
>OPEX for operation and OSS system maintenance, and nonrecurring CAPEX for 
>OSS system.

         I am not sure of that based on my experience with my customers who 
have
deployed 2547 networks.  Certainly CAPEX is required for the equipment to begin
with, but OPEX is actually quite low for 2547 as compared to many other VPN
technologies, especially if your OSS and network deployment strategy is sound
to begin with. I am not saying that management of any network and certain 
2547-based
is easy, but I totally disagree with the assertion that a 2547 network is 
not scalable
based on how it must be managed.

         --Tom

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 23 Nov 2002 10:34:35 +0200

Yakov Rekhter writes:

 > With provider-provisioned Layer 3 VPNs a provider *by definition of
 > the service* has to participate in each VPN customer's routing.  This
 > is irrespective of whether one uses 2547, Virtual Routers, or
 > (provider-provisioned) IPSec between CEs. So, the complexity of Layer
 > 3 VPN service has nothing to do with MPLS, but has to do with the
 > service itself.

yes, and that is why i prefer layer 2 vpls service, where the provider
doesn't need to know anything about the subscribers layer 3 protocols or
routing.

 > routing, the provider could either (a) offer no VPN services at all,
 > or (b) offer Layer 2 VPN services (e.g., FR, ATM, VPLS, etc...).  All
 > of the Layer 2 VPN services I mentioned could be offered with
 > offer Layer 2 VPN services) would not have the "scaleable
 > manageability" problem either.

yes, and therefore i prefer l2tp over mpls.

 >    advantageous to make use of BGP Route Reflectors [BGP-RR] to
 >    improve scalability.  All the usual techniques for using route
 >    reflectors to improve scalability, e.g., route reflector
 >    hierarchies, are available.

how does route reflectors work and improve scalability in multi as/isp
environment?

-- juha