Subject: RE: ISPs offering VPN service

From: "Jesús Rodríguez Stuart" <[email-masked]>
Date: Mon, 4 Sep 2000 19:24:36 -0500

Juan

Actually, Telefonica del PERU is offering VPN Services based on MPLS.

JRSTUART
[email-masked]
Network Solutions Chief - CCEE
Telefonica del Peru
  -----Mensaje original-----
  De: Juan Diego Otero <[email-masked]>
  Para: MPLS WG <[email-masked]>
  Fecha: Domingo, 03 de Septiembre de 2000 07:13 p.m.
  Asunto: ISPs offering VPN service

  Hi, 
  I would like to know if there is any ISP currently 
  offering (or if it is planning to do it) 
  a VPN service based on MPLS. 

  Thanks a lot 

  Diego Otero

---

From: Randy Bush <[email-masked]>
Date: Mon, 04 Sep 2000 17:43:08 -0700

ok.  i have to ask.

we have one early vpn customer who alone has over a thousand vpns (lots of
branches).  so, if i have a few hundred thousand 2547 vpns in my network,
now is bgp gonna like this?  how will the pes perform?

is this thing actually gonna scale, or does it look good for six months
and then run out of gas leaving me with a lot of burned customers?

[ and, in case you were gonna ask, i believe ipsec will scale to millions,
  essentially because it follows the internet architectural principle to
  have the smarts at the edges, not the center ]

randy

---

From: "Brijesh Kumar" <[email-masked]>
Date: Wed, 6 Sep 2000 11:28:24 -0400

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of Randy
> Bush
> Sent: Monday, September 04, 2000 8:43 PM
> To: mpls wg
> Subject: RE: ISPs offering VPN service
>
>
> ok.  i have to ask.
>

I think you are probably right. It seems that the model
in RFC 2547 may be flawed because it fails to separate internal
VPN control and configuration within a domain from the routing
control plane primarily designed for carrying external inter-
domain transit traffic. Besides the scalability problems you
mentioned, adding VPN related policies (basically, internal
traffic of a domain) to something that was originally designed
to provide policy control of inter-domain traffic doesn't seem
obviously appropriate. It is not going to help busy network
administrators
who need to manage large number of BGP nodes, and resolve policy
conflicts in
increasingly large and complex networks.

Cheers,

--brijesh
Ennovate Networks Inc.

---

From: Yakov Rekhter <[email-masked]>
Date: Wed, 06 Sep 2000 09:16:02 -0700

Randy,

> ok.  i have to ask.

And I'll try to answer :-)

> now is bgp gonna like this?  

Could you be more specific, please.

> how will the pes perform?

Could you be more specific, please.

> and then run out of gas leaving me with a lot of burned customers?

The scalability analysis is presented in Section 15 of
draft-rosen-rfc2547bis-02.txt. I hope that analysis answers your
question.

> [ and, in case you were gonna ask, i believe ipsec will scale to millions,

Is it just your *belief* that "ipsec will scale to
millions", or do you have some formal analysis to support this.

And, by the way, there are several ways to use IPSec to build VPNs.
Which one you have in mind ?

>   have the smarts at the edges, not the center ]

"the smarts at the edges" is precisely the model used by BGP/MPLS VPN.
So, following your logic, BGP/MPLS VPN "will scale to millions".

Yakov.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 09:50:21 -0700

Hi Randy,

Let me try to add few comments to Yakov's earlier response.
 
> now is bgp gonna like this?

First if this is one customer this typically means one vpn as all
branches will belong to the same vpn. How he requires to interconnect
them is still flexible both full mesh (any talks to any) or hub and
spoke (branches jaust talk to headquarter) is just a matter of one ext
community value configuration. 

Now regarding the scaling issue. First note that the number of vpns does
not really matter at all here. What matters is the number of routes
injected by each vpn member. In your case typically braches have one
subnet so few thousands of prefixes is still not the issue. 

>   how will the pes perform?

During my current deployment experience with multiple customers there
was never an issue with the PE scaling. Why ? - since I have never came
across any VPN site which would not fit to the PE. So as long as this
condition holds when one PE get's full (I mean he memory is getting full
or CPU for what ever reason is dying) one can easily add another PE and
still grow easily. 

Ok now depending what hardware/software you are using you may get some
scaling limitations on vpnv4 reflector simply due to the fact that in
some cases you may find end customers networks messed up big time
without any hope for summarization. So what we can do here is a few
things: reflector partitioning (based on RT value), (including pushing
ORF to PEs), use router with more memory/cpu power or even use zebra.
 
> and then run out of gas leaving me with a lot of burned customers?

For deployed customers I don't think there is an issue. For new
customers down the path when your current capapcity get's busy they
would need to wait before being added. 
 
>   have the smarts at the edges, not the center ]

There are different ways to provide VPNs and all have it's own usage. It
was said a lot of time but ipsec has slight issue with overlay scheme
associated maintenance. Setting thousands of vpns with nice tool may not
be an issue but reconfiguring it based on the customer's open tickets is
a good job security for some time in the NOC. 

In MPLS-VPNs all smarts are on the edge. In fact expcept reflectors
(which could also be on the edge) core has not idea that there is any
vpn at all. One of the biggest advantages except easy
addtion/modifications of membership or policy I find the dramatic
reduction of customer's routing peers all across his network as he will
peer with one (maybe two for redundancy) rather with thousands.

R.

---

From: Eric Rosen <[email-masked]>
Date: Wed, 06 Sep 2000 12:52:06 -0400

Brijesh> I think you are probably right. It seems that the model in RFC 2547
Brijesh> may be flawed because it fails to separate internal VPN control and
Brijesh> configuration  within  a  domain  from the  routing  control  plane
Brijesh> primarily  designed  for  carrying  external  inter-domain  transit
Brijesh> traffic.

No it  doesn't.  It does  use BGP for  both purposes though.  The  fact that
technology which is primarily designed  for one application is also used for
a second application is not generally considered to be a bad thing.

Brijesh> adding  VPN  related policies  (basically,  internal  traffic of  a
Brijesh> domain) to something that was originally designed to provide policy
Brijesh> control of inter-domain traffic doesn't seem obviously appropriate.

The "VPN-related policies" are primarily expressed as filtering on community
attributes, which is actually fits in quite well with normal BGP functions.

Brijesh> Besides the scalability problems you mentioned, 

The  only  scalability problem  Randy  mentioned is  that  as  you get  more
customers, you might have to add more equipment.  I'd like to see the scheme
in which the hardware cost is independent of the load!

Brijesh> It is  not going  to help busy  network administrators who  need to
Brijesh> manage large number  of BGP nodes, and resolve  policy conflicts in
Brijesh> increasingly large and complex networks.

This is  sophistry.  You  could argue against  any VPN scheme  whatsoever by
pointing out that it creates  some administrative burden, when compared with
not having  any VPN scheme at  all.  That's not very  interesting, it's only
interesting   to   compare   two   VPN   schemes,   for   instance   RFC2547
vs.  configuring,  managing,  and   maintaining  hundreds  of  thousands  of
point-to-point tunnels.

---

From: Sitarama R Kalipatnapu <[email-masked]>
Date: Wed, 06 Sep 2000 13:38:09 -0400

On the note of scalability of BGP/MPLS VPNs, adding more hardware
seems to do the trick. But on a close observation it appears, BGP is the
choke point. BGP running in the provider edge seems to peer with the
customer edges and also distribute MPLS labels across PE edges. The
questions are:

1.  Do we have alternate solutions to provide MPLS VPNs without BGP ?
2.  If so are they effective ?
3.  Is it possible to unload the demands on BGP (carrying  MPLS labels )
     in the BGP/MPLS VPNs model ?

Thank You.

--Sitaram

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 10:56:42 -0700

Sitarama,

BGP is not a choke point. First it is already there in all edge routers
in all ISPs I know of and adding the new AF to it does not cause it to
choke in the right implementation. Notice that the "choking" effect all
depends how one sets the timers (per AF of course - so it does not
effect plain IPV4 updates or other bgp IPV4 mechanisms). As a contary to
what you are saying many customers are setting the VPNv4 timers more
aggresively then ipv4 timers to allow equal to IGP convergence and even
that is not causing bgp to "choke" in a production networks.

The scalability of MPLS-VPNs is achived by the fact that BGP is being
used as a tool there so it would be a mistake to even attempt it's
replacement by some other means.

R.

---

From: Sitarama R Kalipatnapu <[email-masked]>
Date: Wed, 06 Sep 2000 15:23:44 -0400

Robert,

Let me rephrase my question regarding scalability.

1. When do you decide to add another PE router, as the number of
    VPNs supported is increasing ?
2. What makes us add a second PE router ?
3. Is this connected to BGP ?

Thanks,

--Sitaram

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 12:36:55 -0700

Sitarama,

>     VPNs supported is increasing ?

Let me say once again that number of VPNs does not trigger addition of
another PE. Why ? Simply because the overhead of vrf is memory
consumption wise not significant. What does matter is the number of
routes your VPNs give you as you need to store them in the vpnv4 bgp
table as well in RIBs & FIBs for given vrfs. So it really depends on the
operator's comfort with a given router to run at X% of memory
utlization. Notice that in moders routers the memory limit and upgrade
flexibility is growing so even that may not be a very significant
factor. 

The same applies to CPU.

> 2. What makes us add a second PE router ?

See above.

> 3. Is this connected to BGP ?

No. In a peering VPN model you need to maintain customer routes. What
ever protocol you would use you will still have a potential of hitting
the hardware limitations at some point and bgp is quite ortogonal to
this. 

> > >      in the BGP/MPLS VPNs model ?

Using bgp to carry labels for vpn routes is totally not an issue as it
just extending the NLRI size by 20 bits without any processing penalty.
BGP is more or less just a vehicle here.

R.

---

From: Sitarama R Kalipatnapu <[email-masked]>
Date: Wed, 06 Sep 2000 16:32:37 -0400

Robert,

I agree with your analysis. But which entity in the router is consuming
the resources of the router ! That I think is VRFs, BGP RIBs and FIBs.
In that sense, we may say, BGP induced chokepoint of the router.

As we add more VPNs the number of VRFs in the router increases,
pushing the demands on memory and CPU. In the case of BGP peering
with customer edge, BGP demands on the router resources is even
higher. As we rely more on BGP to do everything for the MPLS
VPNs, the CPU time consumed by BGP increases. One of my earlier
questions was on the 'BGP induced choke point'.

> > > 3.  Is it possible to unload the demands on BGP (carrying  interior MPLS labels )

Can something be done about it to ease load on BGP ?

Thanks.

--Sitaram

---

From: "Ramana V Gollamudi" <[email-masked]>
Date: Wed, 06 Sep 2000 16:59:59 -0400

In comparison with a "real" private network (say made up of leased lines) one would
expect the route scaling to be addressed by using route aggregration (for example at
OSPF Area border routers if one is using OSPF within the private network). How is this
type of aggregration done in mpls/bgp VPNs?  Am I correct in assuming that this one of
the scaling issues? And how is this to be solved?

Regards,

Ramana

---

From: "Rajesh Saluja" <[email-masked]>
Date: Wed, 06 Sep 2000 17:00:56 -0400

> the scaling issues? And how is this to be solved?

BGP can advertise aggregated routes from a VRF. This is not an issue with MPLS/BGP VPNs.
( the only issue in this case may be double lookup while receiving packets from the core
)
The scaling issue may be with :
- BGP managing a bigger RIB
- BGP maintaining thousands of EBGP peers ( if BGP is run between CE and PE )

Regards,
Rajesh

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 15:34:58 -0700

Ramana,

> the scaling issues? And how is this to be solved?

If you can summarize in IGP in the leased line model you are fine the
same summarization can be done on either on CE or on PE. If your
addressing alows you can do either IGP summarization or BGP aggregation.
So this is no issue. The issue may exist in some cases when even in the
leased line model any summarization is not possible.

R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 20:53:56 -0700

Hi Sitarama,

> I agree with your analysis. 

I am glad.

> But which entity in the router is consuming

First let me clarify that VRFs structures, RIBs & FIBs have nothing to
do with BGP. They do just fine even if bgp process is not running at all
in a stand alone box configuration. And in that case you may go out of
given box resources at the very close point as if you would run bgp on
this box and peer with other PEs/RRs.

BGP is just the transport vehicle for routes to get to those other edge
boxes which do need them. Isn't it the main task of bgp to advertise
reachability information and associated with them policy ? With
multiprotocol extensions this can be done for different kind of
addresses ipv4, multicast, ipv6, E.164 etc ... in our case we are using
it for vpnv4 addresses.

Afterall if you corelate each prefix to a passenger and bgp process as a
flight crew can one blame a pilot or a flight attendent that this is no
enough sits on the plane. Sure flight attended will have a bit of more
work to do but they can not help that the size of the plane is limited
and that some passengers will need to catch another one. (Of course
plane is a router).

> pushing the demands on memory and CPU.

If those VPNs inject a lot of routes yes. If they are just using one
default out to a headquarter I would not say that even with the number
of vrf reaching platform maxinterface limit there is an issue.

> In the case of BGP peering

Notice that inbound processing, maintenance and update generation are
all independent. Without getting into OS design of any of today
available on the market routers and it's cpu power even with a decent
number of PE-CE EBGP sessions we have never seen as cpu hog caused by
bgp. I am sory that I can't quantify any number here but too many
variables are involved in the equation.

R.

---

From: Greg Ketell <[email-masked]>
Date: Wed, 06 Sep 2000 21:35:31 -0700

At 08:53 PM 9/6/2000 -0700, Robert Raszuk wrote:
>default out to a headquarter

The whole reason customers/ISPs want IP/MPLS VPNs to so simplify networking 
requirements.  Assuming one route per site with no any-to-any reachability 
you have just eliminated the entire benefit of this technology.  Why have 
it in this scenario?

The reality is that people want this for any-to-any reachability which 
means multiple routes per VRF.  Now your flight crew is responsible not 
just for the passengers on this plane but on all the other associated 
planes too.

GK

>I would not say that even with the number
> > > > > > > Brijesh> 
> primarily  designed  for  carrying  external  inter-domain  transit
> > > > > > > Brijesh> traffic.
> > > > > > >
> > > > > > > Brijesh> adding  VPN  related 
> policies  (basically,  internal  traffic of  a

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 06 Sep 2000 22:53:55 -0700

Greg,

> you have just eliminated the entire benefit of this technology.  Why have

This scenario was just to demonstrate the casue of router's resources
limits not to suggest any type of deployment scenario.

Now since you have asked the question there is a lot of current
enterprises is useing leased lines technology or PVC for it's own
connectivity. Those usually go from small sites to big centers. Those
customers are so used to this model that that they want to maintain this
model even if they go to mpls-vpns from any service provider. Sometimes
this comes from centalized application, sometimes just from security
point of view. 

That is not too say that I personely like hub and spoke. In fact I do
agree with you that full mesh is the way to go and mpls-vpns easily
allow it.

> just for the passengers on this plane but on all the other associated

Not really. Remember the same bgp process and the same router handles
all vpnv4 trie/tries (depnding on the implementation choice). So this is
still one plane. The difference is that it happens to carry passengers
belonging to closed groups (marked or separated by RD value) which are
not allowed to be mixed with each other.

R.

---

From: Randy Bush <[email-masked]>
Date: Thu, 07 Sep 2000 08:50:50 +0200

> agree with you that full mesh is the way to go and mpls-vpns easily

while i may agree with you technically. enterprises are used to it, and may
be slow to make the cultural change.  so i expect customers to be in a
common process of evolving reconfiguration.

randy

---

From: "Christian Kuhtz" <[email-masked]>
Date: Thu, 7 Sep 2000 11:16:13 -0400

> common process of evolving reconfiguration.

I'm lost with this argumentation.

In MPLS VPNs, default is that traffic starts to automatically flow shortest
path/fully meshed by the very nature of it.  Basically, all sites of a given
customer just attach to a common net.  So, that said, what evolution is there
for customers to go thru?

No "evolving reconfiguration"...

But, if you really want them to, you can just make MPLS VPNs look like hub &
spoke, why you would ever want to force that inside the vpn (you can always do
it on a TE level regardless, if you desire to) escapes me, but it is possible.

Cheers,
Chris

--
Christian Kuhtz, Sr. Network Architect              Architecture, BellSouth.net
<[email-masked]> -wk, <[email-masked]> -hm                       Atlanta, GA
                                                    "Speaking for myself only."

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 07 Sep 2000 08:57:14 -0700

Hi Ramana,

Now we are moving this thread to a deployment arena which I am always
not clear if this is the right mailing list to continue on ... but well
if the subject is interesting and nothing is vendor specific we can just
proceed until somebody says "stop it".
 
> Robert,
> 
> Is the assumption that most of a given VPN routes will be fed into few PE boxes to make
> useful aggregation feasible at that point? 

In fact a few production networks - especially where the aggregation is
really needed (DSL, cable access lines) into mpls-vpns are using this
method of aggregation. The drawback is suboptimal routing but well
nothing comes for free.

> What happens if the VPN feeds a lot of routes at
> different points into different PE edge boxes ( for example - each CE is a 7-Eleven feeding
> only few routes)? The level of aggregation possible at given PE box may not be much in this
> case. Where does aggregation happen in such a case? Or is this scenario unlikely and hence
> needed not be solved?

This is a likely scenario and you are correct you can not aggregate
much. The good news is that if those are branches like 7-eleven they
usually have one subnet so even having thousands of them (no matter to
which vpns they belong) is not so scary. Much more scary are enterprises
with screwed addressing giving you 100,000 routes from multiple points
that you can not summarize. The plane mpls-vpn technology will still
work as long as you have partitioned reflectors, and good enough routers
as PEs. Of course for those cases (which by the way I only saw a few)
you may also use Carrier's Carrier model, or even CCC/AToM/IPSec - all
depending how big is given customer's mesh. In those examples there is
no issue at all with the number of routes injected into your core.

> In the leased line scenario, I am assuming that the VPN customer has insight into the VPN
> topology and hence can connect his "leased lines or PVCs" in some type of order so as to
> have points where aggregation is feasable. In the mpls/bgp case, the customer just connects
> to service, which is good as it simplifies life for the VPN customer, but the provider needs
> to solve this problem.

Agreed.

R.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 07 Sep 2000 11:09:59 -0700

Christian,

Just to clarify one thing ...

> path/fully meshed by the very nature of it.  

In mpls-vpn there is no default. The prefix distribution among PEs via
IBGP fully depends on the policy you configure. There is no default
policy (at least in one vendor I know a few things about :). 

So for one operator full mesh configuration may be the default he has
chosen, for the other hub and spoke with hub as a headquater CEs, yet
for another hub & spoke with hub being a PE (aggregation case discussed
earlier). Good mpls-vpn provisioning tool should allow you to pick any
of the above by just click of the mouse on a per VPN basis.

Based on the real life observations I agree with Randy for the
transition to full mesh for everybody may take time.

R.

---

From: Randy Bush <[email-masked]>
Date: Thu, 07 Sep 2000 20:45:04 +0200

> Much more scary are enterprises with screwed addressing giving you 100,000
> routes from multiple points that you can not summarize.

operationally, we do not usually consider customers with separate address
space at each location to be screwed, and we try not to screw them.

> no issue at all with the number of routes injected into your core.

but i am cheered that you're seeing why i am worried about this scaling.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 07 Sep 2000 11:47:00 -0700

Randy,

> but i am cheered that you're seeing why i am worried about this scaling.

Yes I do. But the worry comes from the currently available routers
resources perspective not the mpls-vpn technology by itself. With a new
router generation where you can store >> then 100,000 routes (comming
soon at least from three vendors I know about) I will not be worried
anymore :).

R.

---

From: Randy Bush <[email-masked]>
Date: Thu, 07 Sep 2000 21:21:12 +0200

> soon at least from three vendors I know about) I will not be worried

the problems with bgp scaling is not a matter of ram.  for a scary example
of one small aspect of the current bgp mess see

    http://www.acm.org/sigcomm2000/conf/paper/sigcomm2000-5-2.ps.gz

randy

---

From: Greg Ketell <[email-masked]>
Date: Thu, 07 Sep 2000 15:29:56 -0700

>soon at least from three vendors I know about) I will not be worried

You should be.  As you pointed out earlier, many customers with many route 
prefixes per customer is the target of this service.  With most of these 
prefixes being aggregated today we have route table growth that is faster 
than some router vendors can keep up with.  Now, unaggregate them all (or 
even just 10-20% of them).  Now you are outgrowing the Processor 
manufacturers and RAM manufactures capabilities.  Yikes!

The fact that adding one prefix on a PE router in timbuktu will affect all 
your other PE routers (either as an additional prefix or as additional 
filtering requirements) and you have a scenario where a router that was 
working fine could go belly up with no one any where near it.

ORF will help the PE but at the cost of performance on the RRvpn's.  So now 
you add more RRvpn's...  Just how good is that provisioning tool anyway?

GK

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 07 Sep 2000 19:44:49 -0700

Hi Greg,

> prefixes per customer is the target of this service.

Many customers with many prefixes is still fine even today. Many
customers with HUGE amount of unaggregated prefixes is getting us close
to the hardware limits - but still on a per box basis - not per mpls-vpn
as a whole.

There are two reasons why I am not too worried about the extreme cases
are:

A. I know only 3-4 worldwide big enterprise networks which may be
getting to this kind of prefix numbers in a range of 100,000 routes in
their own internal network.

B. Those from item A I talked with were going to use mpls-vpn by
themselves as they already running the entire network by themselves to
easily connect their department and branches and would not be intersted
any time soon to give the function of running the corporate network to
SP/ISP.

So what it means to us is:

1. The cases where we hit the scaling limits of today's hardware will be
not too often
2. If those global enterprises will start looking at MPLS from the
ISP/SP perspective Carrier's Carrier solution    of mpls-vpn will in a
scalable way alow to accomodate such customers into any provider's core.

> working fine could go belly up with no one any where near it.

Ha that I would call very poor implementation of mpls-vpns. The one I am
familar with a bit does not have this characteristic :) There are very
easy ways to avoid this from happening, otherwise it would be quite a
challange to keep the IPSs/SPs network running at all provided that all
prefixes are injected dynamically.
 
> ORF will help the PE but at the cost of performance on the RRvpn's.  

This is in fact just the opposite :). You want to limit the routes each
reflector receives to only those he is interested in serving by pushing
policy to PEs. PEs do the filtering before sending vpnv4 updates so only
those updates which needs to be received by given reflector cluster are
being send accordingly. This helps in two ways: a) frees up relfectors
from additional inbound filtering, B) distributes the filtering to many
PEs - "smart edge" concept.

Since you have asked - on the reflectors on the other hand all PEs can
be in a peer group so update generation happens only once and then is
just the low cost replication.

R.

---

From: "dave o'leary" <[email-masked]>
Date: Thu, 07 Sep 2000 22:14:51 -0700

>policy (at least in one vendor I know a few things about :).

I think that Christian's point is that if no policy is explicitly configured
to filter (whether directly in the routers or via a provisioning system),
then it is reasonable to assume that the VPN attributes will be distributed
around the full mesh just as regular IPv4 prefix information is distributed,
hence a full mesh VPN.  At least in my recollection of configuring one
vendor's products (the same one that I assume you are talking about),
when (i)BGP peering is established, the default is to send everything
unless it is explicitly configured.  Of course, it has been a few years
since I've configured BGP on a type-C box.... :-)

                                                 dave

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 07 Sep 2000 22:34:14 -0700

Dave,

Actually distributing prefixes everywhere and even worse accepting them
to all PEs would be very bad idea in the mpls-vpn case. The "timbuktu"
example Greg brought earlier should tell you why.

So by default (meaning without configuring any policy = without setting
any route target ext. community) vpnv4 routes should not be accepted to
any of the remote vpnv4 bgp tables then vrfs and hence by default you
will see no routes in that vrf  and (as Eric Gray correctly pointed out
offline) by default you will see packet drop :).

As this is implementation specific issue not related to interoperability
I don't think that we should continue on this particular topic on a
mailing list dedicated to mpls protocol specific issues. Of course other
mpls-vpn architecture related questions are still welcome.

R.

---

From: Eric Rosen <[email-masked]>
Date: Fri, 08 Sep 2000 10:11:32 -0400

Randy> for a scary example of one small aspect of the current bgp mess

While it may be true that Internet  routing is a mess, it is disingenuous to
blame that on  BGP.  BGP is the routing technology  that allows the Internet
to scale as well as it does  (which is pretty damn well), and nothing better
is known.

The "messiness" is not due to any inherent characteristic of BGP, but to the
messiness of the Internet routing environment itself, with so many providers
who don't trust each other and can barely cooperate.  BGP did not create the
mess, it is just the best way  we know of coping with it.  There's certainly
no  reason  why  BGP should  not  be  used  to  advantage  in a  less  messy
environment,  it is  after all  the  most scalable  routing protocol  known.
(Outside of the research community, at least.) 

But as is clearly stated in RFC2547, it is not aimed at supporting VPNs over
the  Internet, so  the  messiness  of the  Internet  routing environment  is
irrelevant.

---

From: Eric Rosen <[email-masked]>
Date: Fri, 08 Sep 2000 10:30:57 -0400

Ramana> What happens  if the VPN feeds  a lot of routes  at different points
Ramana> into different PE  edge boxes ( for example - each  CE is a 7-Eleven
Ramana> feeding only few routes)?

There  are  a  number  of  techniques  which can  be  used  to  improve  the
aggregatibility.  In  this particular  case, I think  it unlikely  that each
of the  100,000 7-11 stores connects  to a different PE  router.  Each store
probably connects directly to an  access network of some sort which attaches
a whole bunch of  stores to a single PE router, where  they can all be homed
to the same VRF.

---

From: Eric Rosen <[email-masked]>
Date: Fri, 08 Sep 2000 10:38:10 -0400

Randy> then why are we discussing it  in the ietf?  i guess we should should
Randy> have the rfc made irrelevant.

It is being  discussed in the IETF  because (a) there are a  large number of
service providers who find the  technology useful, (b) the service providers
want  public  standards  to  ensure  multi-vendor  interoperability  of  the
technology, (c)  the technology is IP-based,  and (d) the IETF  is the place
for setting standards based on IP technology. 

However, you may wish to take this up further with the Area Directors.

---

From: "Thomas Huang" <[email-masked]>
Date: Sat, 18 Sep 1999 08:14:25 -0700

Hello:
I noticed mpls-te@uunet also appeared in some of the email here.

Is there a mailing list discussing the MPLS-TE more intensively?
If so, where could I register on?

Thanks.
/Thomas

---

From: "Ramana V Gollamudi" <[email-masked]>
Date: Fri, 08 Sep 2000 11:38:59 -0400

The 7-Eleven (large VPN customer) is only one case that could lead to this
scenario of route scaling. There are other ways this could happen, but I agree
that having  lot of CEs terminated on a few PEs is one way of solving the route
scaling problem. Given the existing infrastructure, this is likely to be the
common way. Another extreme example of solving the route scaling problem for
large VPN customers would be to back haul all the traffic of large customers
into few large PE routers which are connected in some private mesh using
VCs/LSPs, and avoid a lot of the complexity associated with pushing all these
routes into BGP and/or internet stability etc. altogether. To me this implies
two assumptions in this type of VPNs, there is a  tunnel (leased line/ATM or FR
VC/LSP) type access network and large capacity PE boxes, as opposed to starting
with small boxes and adding more  boxes with growth. True?

Regards,
Ramana

---

From: Randy Bush <[email-masked]>
Date: Fri, 08 Sep 2000 16:19:12 +0200

> But as is clearly stated in RFC2547, it is not aimed at supporting VPNs
> over the Internet, so the messiness of the Internet routing environment is
> irrelevant.

then why are we discussing it in the ietf?  i guess we should should have
the rfc made irrelevant.

randy

---

From: Sitarama R Kalipatnapu <[email-masked]>
Date: Fri, 08 Sep 2000 17:58:35 -0400

I will have to agree with Eric Rosen on this. BGP is the only way
inter-domain routing can be achieved in the Internet and that is
what it is designed for. BGP cannot be blamed for this
'role of BGP'. Can something be done about it !

However the issue here seems to be the other roles that BGP
plays in the context of MPLS VPNs and its consequential
effects. May be something can be done here.

--Sitaram

---

From: Randy Bush <[email-masked]>
Date: Sat, 09 Sep 2000 10:35:52 +0200

> having  lot of CEs terminated on a few PEs is one way of solving the route
> scaling problem.

in a very large or global isp?

randy

---

From: "Ramana V Gollamudi" <[email-masked]>
Date: Mon, 11 Sep 2000 09:54:58 -0400

Randy,

This is one of the ways that was suggested to get around the problem that occurs
when route summarization becomes a problem. If you are concerned that in global
ISPs this back hauling of traffic over VC technologies is not
operationally/economical feasible then certainly the problem remains, and it
would help if more operators spoke up on the need to solve this aspect of the
problem.

---

From: Randy Bush <[email-masked]>
Date: Mon, 11 Sep 2000 15:48:54 +0200

> This is one of the ways that was suggested to get around the problem that
> occurs when route summarization becomes a problem. If you are concerned
> that in global ISPs this back hauling of traffic over VC technologies is
> not operationally/economical feasible then certainly the problem remains,
> and it would help if more operators spoke up on the need to solve this
> aspect of the problem.

what does the backhauling do to my customer's latency (and when latency goes
up, so does the likelihood of increased jitter)?  was the additional latency
what they expected from a vpn service?

randy

---

From: "Brijesh Kumar" <[email-masked]>
Date: Mon, 11 Sep 2000 10:41:45 -0400

Hi Ramana,

I don't think that is the way scalability problems with RFC 2547 can
be addressed.

It is not practical to terminate lots of CEs over a single PEs. More
interfaces means more PE-CE routing adjacencies. By doing what you
suggested, you will only shift PE-PE domain scalability issues to
PE-CE domains. Making a PE more powerful than they really need to be,
and forcing lots of CEs to peer with some limited number of PEs, one
will significantly add to the cost of infrastructure. The good
solution is the one that does not put impractical constraints on
carriers for deploying services.

Does I-BGP to I-BGP communication model provide an acceptable solution
for VPN signalling in terms of scalability, granularity, speed of
convergence and ease of maintenance? Many don't believe it does!

Cheers,

--brijesh
Ennovate Networks Inc.

> From: [email-masked] [mailto:[email-masked]]On Behalf
> Of Ramana V
> Gollamudi
>
> Randy,
>

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 11 Sep 2000 10:52:09 -0400

> a whole bunch of  stores to a single PE router, where  they can all be homed

Last I remember configuring routers for a provider aggregation had to
be configured (they don't happen automagically) and generating
aggregates from a database was a black art with too few practitioners.
In fact configuring anything from a database was practiced by too few
practitioners partically leading to this Internet routing mess that
we've wasted so much time discussing.

Managing the aggregation of private address space for the benefit of
each customer is something I'd rather not have to do, but then again I
don't work for a provider anymore so I guess someone else gets to
decide if this sounds like too much work.

Then again, giving the customer a static route and a single (easily
aggregted) prefix at each interface and requiring the customer to do
multihop BGP to get any private addresses would probably work fine,
scale fine, and only look slightly ugly architecturally.  Then only
the CPE router is concerned with any prefix explosion in private
address space.

Over and out.

Curtis

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 11 Sep 2000 08:54:23 -0700

Brijesh,

What are PE-PE domain scalability issues with 2547 ? This is not what we
were talking about at all here.

R.

---

From: Eric Rosen <[email-masked]>
Date: Mon, 11 Sep 2000 12:31:57 -0400

Brijesh> It is not practical to terminate lots of CEs over a single PEs. 

On  the contrary,  each  PE generally  attaches  to lots  and  lots of  CEs.
Provider edge  systems tend  to be  quite powerful, and  have lots  of "high
touch" features. 

However,  the aggregation  problem for  VPNs is  not quite  the same  as the
aggregation problem  in global  routing.  In a  default-free zone,  a global
routing table (for the public Internet) must hold (non-default) routes which
match ALL  globally reachable addresses.  A VPN-specific  routing table only
needs to hold  routes for addresses within the VPN.   No single router needs
to hold routing  tables for all VPNs.  So the pressure  to aggregate the VPN
addresses is not  nearly the same as the pressure  to aggregate the globally
reachable  addresses.  For  most VPN  customers, this  won't be  much  of an
issue.   In  practice,  I  think  that  this is  a  major  issue  only  when
integrating dialup connections and the like into the VPN.

Curtis> Managing the aggregation of private address space for the benefit of
Curtis> each customer is something I'd rather not have to do, but then again
Curtis> I don't work for a provider  anymore so I guess someone else gets to
Curtis> decide if this sounds like too much work.

One needs  to keep in mind that  the provider might otherwise  be managing a
large number  of point-to-point VCs,  and then providing a  managed backbone
service to  each customer as well.  As  I've said before, it's  not valid to
compare the amount of work needed  to provide VPN service with the amount of
work needed if you don't provide it.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 14 Sep 2000 09:02:02 -0400

> work needed if you don't provide it.  

To configure a VC you don't need to know about the subnets behind the
customer's CE router.  

You seemed to have lost my point.  If you need to configure things on
the PE to help manage the customers proviate address space the amount
of work can become unbounded (largely due to the fact that the amount
of customer clue is highly bounded, ie "what's CIDR mean").  I
suggested a way to avoid participating in managing the customers
private address space.  That would make the work comparable to that of
managing VCs as long as the public side addresses were sensibly CIDR
allocated.

Curtis