Subject: RE: Scalable and manageable network management and operation of M PLS/VPN

From: "Chen, Weijing" <[email-masked]>
Date: Fri, 22 Nov 2002 12:50:12 -0600

Robert,

Thanks for reply.  However, I don't think that I mixed the VR-based and
2547bis-based VPN.  Regardless it is a VR or a VRF, from management
standpoint, it is a router that we must management, e.g. route target id,
import id, export id, etc. in 2547bis case.  Also for VRF and PE connection,
LSP labels in both end and sometime LSP labels in the middle.  As long as
they are something that we must establish or assign, keep track of, and
troubleshoot, they require system resource and human resource.  And those
resources come with cost.

It reminds me of ATM (S)PVC.  Back then, all the (S)PVC people assure us
that it is not big deal since it is just a connection.  10 years from then,
we are still bearing the pain of provisioned connection-oriented (S)PVC.

IP is great since it is connectionless.  Telephone (POTS) is also great
since it is connection-oriented with fully functioned end-to-end subscriber
initiatied signaling.  Either one will work for us, but not something in
between.  Oh, well, I went too far.

--
Weijing Chen
SBC Technology Resources
9505 Arboretum Blvd.
Austin, TX 78759
512 372 5710
[email-masked]

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Friday, November 22, 2002 12:24 PM
To: Chen, Weijing
Cc: 'Yakov Rekhter'; 'PPVPN'
Subject: Re: Scalable and manageable network management and operation of
MPLS/VPN

Weijing,

Your below email clearly indicates that you are mixing VR based L3VPNs
with the L3VPNs described in 2547bis architecture as non of your points
apply to the latter one. I must admit that they are very true for the
first type of L3VPNs though :-).

Thx,
R.

> Yakov,
> 
> 
> 
>    Sorry for late response. I didn't receive your reply for somewhat
reason. Yetik pointed this message to me and I found it from mailing list 
> archive.
> 
> 
> 
>    >>Could you please elaborate on what exactly you are concerned with
respect to "scaleable manageability of MPLS/VPN"?
> 
> 
> 
>      In a nutshell, RFC 2547 doesn't scale because it breaks the rule that
all we need to do to manage a subscriber is manage the 
> attributes on the port serving the subscriber.  We have to dedicate
resources to the subscriber that go deeper into the network than just 
> their port.  Establishing, keeping track of, and troubleshooting these
resources on a per-subscriber basis will be difficult.
> 
> 
> 
>      Specifically, the resources we have to dedicate and manage are:
> 
>    #Virtual Routing Functions.  Nearly one for each CE/PE interface.  Each
one of these, from a management standpoint, is a 
>      router.  Right now, large IP carriers manage hundreds to possibly a
few thousand routers.  With IP VPNs, we'll be managing tens of 
>      thousand to possible hundreds of thousands of routers (VRF), a good 2
orders of magnitude jump.
>    #Connections between VRFs.  Each VRF in a VPN has to have a connection
to every other VRF in the VPN.  The more VRFs 
>      you have (100,000), the more connections you have between them.
>    #Connections between PEs.  Each PE in a VPN has to have a connection to
every other PE in the VPN.  This connection 
>      maybe shared by multiple VRFs in the same PE pairs.  However,
tracking of what connection to be shared or establishing new one if 
>      not shared is another daunting task.
> 
> 
> 
>      Each single item above is translated into $$, which including
recurring OPEX for operation and OSS system 
> maintenance, and nonrecurring CAPEX for OSS system.
> 
> 
> 
>    >>Also, when you said "we operation group in service provider", do you
mean a specific service provider?
> 
>    Any service provider with hundreds of POPs (or COs), thousands of PEs
(routers or switches), millions of 
> subscribers (number of sites), such as us
> 
> 
> 
>    Regards,
> 
> 
> 
> 
> 
>    --
> 
>    Weijing Chen
> 
>    SBC Technology Resources
> 
>    9505 Arboretum Blvd.
> 
>    Austin, TX 78759
> 
>    512 372 5710
> 
> 
> 
> 
> 
>

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 22 Nov 2002 20:33:49 +0100

Weijing,

All I can add to my previos msg is to indicate that it really takes a
bit more to manage a router then assinging RD & RT in 2547bis. I wish
routers would be so smart to just take two values and operate ;-). And
to remind you it is all (along with applying PE-CE routing exchange
template) what addition of new VPN site requires from your managment
station. 

Also reg the LSPs as a transport notice that one may run 2547bis without
any MPLS LSPs at all. Appriopriate drafts discussing how to do this have
already been posted to this group. Oh and btw ask your vendors what are
pros & cons of not using MPLS LSPs as your transport vehicle for VPN
packets.

Rgs,
R.

> Either one will work for us, but not something in

PS. IMHO LDP LSPs are connectionless, while TE LSPs connection-oriented.
I don't really see anything in between here :).

> "Chen, Weijing" wrote:

---

From: Thomas Nadeau <[email-masked]>
Date: Fri, 22 Nov 2002 15:09:49 -0500

>resources come with cost.

         Of course, the management of any new service is going to
cost additional money. However, I think that we need to be clear on
just how much. One of the advantages to MPLS L3VPN is that it
can be managed as part of your overall MPLS management strategy
given that it is an extension of MPLS. I think that making a statement like
"it is not scalable because it cannot be managed effectively" is at best
inaccurate.

         --Tom

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 23 Nov 2002 10:40:35 +0200

Thomas Nadeau writes:

 > One of the advantages to MPLS L3VPN is that it
 > given that it is an extension of MPLS. 

in case the provider has no other use for mpls than vpns, then the
provider needs to start managing mpls in addtion to managing ip, which
clearly is an additional burden.

-- juha

---

From: Chris Chase <[email-masked]>
Date: Sat, 23 Nov 2002 10:18:41 -0500

I don't normally follow this mailing list, but an associate pointed me 
to this thread.  Sorry to jump in the middle.

Inline.

>we are still bearing the pain of provisioned connection-oriented (S)PVC.
You are equating resources with cost and I assume you are saying that 
"pain" for VC service is in terms of costs.  So what are you concluding 
here about PVC service?  Has it been a pain for SBC?  SBC has the 
largest U.S. FR market share of the ILECs (although much smaller than 
the IXCs), but approaching 1/3 billion (market analyst reports for 
2001).  Are you saying it is so much of a pain (in terms of costs) that 
SBC can not make money with it?  I know that for AT&T and most other 
carriers PVC service is very profitable.  I suspect the same is true for 
SBC - you may want to talk to your SBC associates that deliver those 
services.

Certainly there are complexities and resource consumption with creating 
these network based VPN services, whether Layer 2 or Layer 3.  But is 
that not the value add we as carriers bring to the equation?  We provide 
the value add of these services for enterprise WAN service while making 
money because we do this at scale.  Otherwise, you would not being 
seeing enterprise WANs migrating off private lines nor carriers 
reporting profitable business data services.

Carriers have found how to scale the network based layer 2 services to 
tremendous sizes and be very efficient with running these services. We 
believe we can do (and are doing) the same with the nascent Layer 3 (IP) 
network based VPNs.

>between.  Oh, well, I went too far.
Layer 3 network based VPNs are connectionless IP from the enterprise WAN 
user view.  Layer 2 network based VPN are connection oriented and proven 
from the enterprise WAN user view.   When you say these services (by 
implication the "something in between") will not work for "us", who is 
"us" - the SBC point of view or the corporate enterprise WAN view?  

Chris Chase
AT&T Labs Research

---

From: "Brijesh Kumar" <[email-masked]>
Date: Sat, 23 Nov 2002 13:42:27 -0800

Chris,

I guess, it was your conclusion that Yakov quoted  as 2547 BGP/MPLS VPNs
being "Certainly scalable and solvable.".  I am not doubting that assertion,
but I would like to know a bit more. I am wondering what is the scale, we
are talking about here? 100s of VPNs, 1000s of VPNs per PE?.  How many sites
per VPNs? in tens or hundreds?

There is yet another issue on which I would like to seek an answer from
people like you who seem to have operational experience with configuring and
managing Layer 3 VPNs in live networks.

There are two aspects to VPN service scalability - the router engineering
aspect, and the operational aspect. The first aspect does not look like an
issue  as the current generation of equipment is designed (or claimed) to
facilitate carriers to provision several thousand Layer 3 VPNs (regardless
of the VPN model used), if not tens of thousand VPNs like existing Layer 2
technology. I don't know if any one can guarantee that with their equipment,
a carrier can support a few thousand VPNs with  a few hundred sites per VPN,
but most can do a few thousand VPNs with moderate number of sites (<20). I
would like to know if any one is running Layer 3 VPNs that have sites in
three digits.

If I understood  Wienjing's mail correctly, he seems to be pointing to the
lack of operational scalability i.e., the lack of suitable VPN management
and configuration tools that allow operators to manage a large number of
layer 3 VPNs, taking a network wide view (as opposed to a point-by-point, PE
oriented view). Do you believe that current generation of operation support
software provided by your vendor X is good enough for operators to run
large networks with thousands of VPNs with some VPNs having hundreds and
possibly a few thousand of sites?

Cheers,

--brijesh

----- Original Message -----
From: "Chris Chase" <[email-masked]>
To: "Chen, Weijing" <[email-masked]>
Cc: <[email-masked]>; "'PPVPN'" <[email-masked]>
Sent: Saturday, November 23, 2002 7:18 AM
Subject: Re: Scalable and manageable network management and operation of M
PLS/VPN

---

From: "Jonathan Jiang" <[email-masked]>
Date: Sun, 24 Nov 2002 17:55:35 -0500

Brijesh,
Regarding tools for configuring and managing MPLS VPN, I am aware of at
least three OSS products that aim to manage thousands of VPNs, each with
hundreds of sites.  While they are not in wide deployment yet, I believe
it's just a matter of time.

Thanks
Jon

-----Original Message-----
From: Brijesh Kumar [mailto:[email-masked]]
Sent: Saturday, November 23, 2002 4:42 PM
To: Chris Chase; Chen, Weijing
Cc: 'PPVPN'
Subject: Re: Scalable and manageable network management and operation of
M PLS/VPN

Chris,

Cheers,

--brijesh

----- Original Message -----
From: "Chris Chase" <[email-masked]>
To: "Chen, Weijing" <[email-masked]>
Cc: <[email-masked]>; "'PPVPN'" <[email-masked]>
Sent: Saturday, November 23, 2002 7:18 AM
Subject: Re: Scalable and manageable network management and operation of M
PLS/VPN

---

From: Mudhafar Hassan-Ali <[email-masked]>
Date: Sun, 24 Nov 2002 17:30:07 -0800

Jon,

It would be useful if you could tell us who they are.

Regards,
Mudhafar

---

From: Chris Chase <[email-masked]>
Date: Mon, 25 Nov 2002 14:26:38 -0500

inline

>per VPNs? in tens or hundreds?
I think he referred to a talk I gave last year.

Okay, first to set the scope here, we are talking about network based 
MPLS VPNs for Corporate intranets, not VPNs for remote or home office 
access (which is yet another management problem - to plug those into the 
network based MPLS VPN).

Eventually I would hope 1000s of VPNs.  Sites in VPNs: dozens up to 
10,000, but the distribution will fall off rapidly.  There just aren't 
that many companies with more than a 1,000 sites compared to the many 
smaller enterprises with dozens or a couple hundred sites.  VPNs of size 
thousands are large retail business enterprises (we have these.  Of 
course there are significant numbers of these types of enterprises using 
private line or FR/ATM service for their corporate intranets, the 
largest of which I have seen is in the multiple 10,000s).

Per PE requirements is not a simple question.  The objective is that we 
efficiently use the PE capacity and obtain a reasonable per port cost. 
 We do not yet know if we can get a PE that has multi-gigabit per slot 
access capacity and be able to terminate low speed (T1 or lower) access 
efficiently while running routing and QoS queues for every one of those 
ports.  So there can be different PEs for different domains.  

For enterprise networks port speed distributions are very heavy weighted 
on the low speeds when you get into more than a few dozen sites.  They 
could have hundreds of low speed sites (T1 or lower) and a handful of 
high speed sites (T3 and higher).  So for a PE that terminates only high 
speed (>= T3), every high speed port is likely a unique VPN, but then 
you only have ports numbering in the hundreds for current potential PEs. 
 For a low speed PE, you are likely to have multiple of connections per 
VPN, so while you may have potentially a couple thousand ports you will 
have an order magnitude less VPNs (or you might even have a smaller 
switching capacity PE with only hundreds of ports and then even lower 
VPN numbers).

>would like to know if any one is running Layer 3 VPNs that have sites in
We have a large number of VPNs in the three digits.  We even have a VPN 
in the multi thousands.

The PE router engineering is straightforward.  Most of the provisioning 
problem is with the access circuits and termination.  At our peak growth 
we were provisioning over 2,000 customers per week with the FR service. 
 We do have that growth rate with VPNs yet, but we hope to get there.  I 
don't really see a problem with that.  The layer 3 VPNs are more of a 
challenge to get right because there are more orderable parameters 
(layer 1+2+3; policing, shaping, queuing) than we had with layer 2 or 
layer 1 services.  The more paramters the more opportunities for 
problems, the more things to troubleshoot.  But we have flow through 
provisioing today (no human hands from order entry to PE service creation).

>possibly a few thousand of sites?

Provisioning scale we have.  I won't make a statement about existing 
tools in the market place.  We tend to have our own rolled OSS's.  We 
often use off the shelf management products, but more in a plugin (like 
a library, toolkit or module) than a complete system that does ordering, 
inventory, workflow, billing, alarming, ticket tracking, stats 
collection, reporting, etc.  I have never believed in these wonder 
systems that claim to do it all.  They never meet all our specific 
requirements.

I do think there are operational challenges with Layer 3 versus Layer 2 
VPNs for customer facing problems.  With Layer 3 VPNs the carrier is 
participating in the customer's routing.  For all the supposition of 
connectionless IP service having less "pain" [as implied in the original 
poster] it is more customer support intensive.  And thus we need more 
sophisticated troubleshooting tools.  Layer 1 and Layer 2 services have 
very clear demarcs.  For example, internally, the network providing 
layer 2 service may be complex, e.g., running PNNI, but the customer 
service is fairly straightforward -- does LMI say the PVC is up?  is my 
access circuit up?  Is my bandwidth less than contracted?  You don't get 
IP routing trouble tickets, but in layer 3 routing blurs between the 
provider and the customer.  Actually, it is because managing and 
supporting Layer 3 IP services are more complex that providers hope to 
make a lot of money by managing that for the enterprise.

Chris Chase
AT&T Labs Research

>>>import id, export id, etc. in 2547bis case.  Also for VRF and PE
>>>      
>>>
>connection,
>  
>
>>>that it is not big deal since it is just a connection.  10 years from
>>>      
>>>
>then,
>  
>
>>>since it is connection-oriented with fully functioned end-to-end
>>>      
>>>
>subscriber
>  
>
>>>>    In a nutshell, RFC 2547 doesn't scale because it breaks the rule
>>>>        
>>>>
>that
>  
>
>>>>        
>>>>
>>>>  #Virtual Routing Functions.  Nearly one for each CE/PE interface.
>>>>        
>>>>
>Each
>  
>
>>>>        
>>>>
>>>>    thousand to possible hundreds of thousands of routers (VRF), a good
>>>>        
>>>>
>2
>  
>
>>>>        
>>>>
>>>>  #Connections between VRFs.  Each VRF in a VPN has to have a
>>>>        
>>>>
>connection
>  
>
>>>>        
>>>>
>>>>  #Connections between PEs.  Each PE in a VPN has to have a connection
>>>>        
>>>>
>to
>  
>
>>>>        
>>>>
>>>>  >>Also, when you said "we operation group in service provider", do
>>>>        
>>>>
>you
>  
>
>>>>        
>>>>

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Mon, 25 Nov 2002 15:15:53 -0500

>clearly is an additional burden.

         Yes, that is true and depending on your new management
approach, your mileage may vary. However, the typical deployment cases I
have seen are where MPLS VPN is added to an existing MPLS-enabled
network, therefore adding the addition of MPLS VPN management to
the existing OSS is evolutionary.

         --Tom

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Mon, 25 Nov 2002 15:26:04 -0500

>possibly a few thousand of sites?

         As with any technology, there are lots of different approaches
to how to manage that technology that vary widely from totally off-the-
shelf  purchased software, to solutions developed totally in-house.
In terms of applications available to manage MPLS/VPN, many commercial
3rd party applications exist to many some or all aspects of MPLS/VPN
networks, as well as those produced by vendors specifically for their 
products.
In many cases, the applications are built upon those that can manage other
MPLS applications, given that it makes them both more modular, and more
suitable for different operating environments.  I will leave it to the 
operators
to speak to the effectiveness of their particular approach, but I can say that
numerous (very scalable) approaches to management do exist and are in use
today.

         --Tom

> > >>subscribers (number of sites), such as us

Success is relative; the more success, the more relatives. -Anonymous

---

From: "Brijesh Kumar" <[email-masked]>
Date: Mon, 25 Nov 2002 13:44:02 -0800

> 3rd party applications exist to many some or all aspects of MPLS/VPN
<snip> text deleted <snip>
> butI can say that

Thomas:

Can you please be more specific which particular third party OSS tools you
are referring to, and what specific features they have for scalable Layer 3/
(Or MPLS/VPN) deployements?  I think someone else also asked this, but
didn't get any response.

Cheers,

--brijesh

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Mon, 25 Nov 2002 18:06:50 -0500

>(Or MPLS/VPN) deployements?  I think someone else also asked this, but

         This mailing list is not appropriate for vendor-specific products.
I suggest that you move this to the mplsrc mailing list where those
vendors may respond personally.

         --Tom