Subject: Re: [pim] Fwd: Multicast in MPLS/BGP IP VPNs

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2005 09:38:53 -0700

>> If you don't think PE-based VPNs will scale for multicast services, please can you explain how you think a CE-based VPN approach will scale better? I would be particularly interested in hearing how you think bandwidth consumption will compare using the CE-based approach, bearing in mind that all replication will need to take place at the CE, i.e. for N receivers the source CE access link will need to carry N copies of the same packet.

    Mike McBride and Yiqun Cai will present their draft at the L3vpn session.

>> All solutions/technologies have scaling limits that when reached processing/bandwidth resources need to be increased or an alternative solution/technology needs to be selected. Please can you elaborate as to what the scaling limits are when using PE-based VPNs that cannot be overcome by increasing network resources incrementally as the number of customers increases, e.g. adding new route reflectors? 

    Yes, you are right. Increasing resources costs money. There are times 
    when you just can't solely throw hardware resources at a problem.

    In a CE-based solution, the hardware resource is added for the new site
    (if not already there) and the PE doesn't need add anything.

>> I assume here you are talking about overlaying multicast VPNs over PE-based unicast VPNs? 

    Yes.

>> > PE-based VPNs will not scale, it is the wrong place to put the technology.
>> 
>> Really? I think there may be a few operators that will disagree with you. Again, please can you explain what the scaling limits are and why they can't be overcome by increasing network resources?

    Yes, they will disagree but not for this reason.

>> So?.. Distributed scaling is not unique to the CE based approach, PE based VPNs can also scale in a distributed manor. For example, route reflectors can be added as the number of customer routes increases and new blades can be added to PE chassis incrementally as customer demand increases.  

    Distribute even more, closer to the edges so the CE boxes handle only the
    VPN it is attached to. And you get inter-AS VPNs for free.

    Do you realize how complicated intra-AS MPVNs are, let alone inter-AS,
    and extranet based multicast services?

>> Please can you explain how the CE-based approach is more cost-effective than the PE-based approach? You don't need to pay for a "huge honkin PE box" from the outset. PE chassis are relatively cheap and blades can be added incrementally as customer demand increases. Putting device equipment costs aside, how do you think bandwidth costs/efficiency for multicast traffic compares using the CE-based approach (where N streams will need to be carried for N receivers on the source access link)? 

    No, you buy a PE box that aggregates connections. That is the physical
    requirement. But adding all the policy-basd edge features and complicated
    control-plane interaction which operators have to deal with (read: high
    OpEx) then the cost of the box and the service goes up. And as you add
    customers and more VPNs, and then more services on top of it (read: 
    multicast) it isn't going to get easier.

>> Regarding privacy, operators have spent a significant amount of time and money investing in network provisioning/management systems to ensure VPN isolation is maintained. If you believe privacy is an issue, what do you think the issues are? Regarding single points of failure, failure of a CE connected to a central office or hub site in the CE-based VPN approach is just as detrimental to the customer as a PE failure in the PE-based approach. In addition, in the CE based approach a CE still needs to connect to an edge router in the providers network, which without any redundancy will be a single point of failure. The only way to protect against failure is to use redundancy (in the form of devices, routes, links, cards, etc.), this is orthogonal to whether a CE or PE based VPN approach is used.

    VPNs are hyped as being "private". But private to me means my data is
    encrpyted before I hit the ISP (I actually prefer my data encrpted before
    it leaves the system I am typing on). But VPNs really deliver "separation"
    and very loosely at that.

    I have also heard that VPNs can be made secure by justing using IPsec
    end-to-end. Well I can do that without VPNs as well. So who is helping
    who here to provide the privacy?

>>Who says that? Providers offer managed and unmanaged VPN solutions (where management refers to the management of the CE) so the provider may need to manage the CE whether a CE-based or PE-based approach is being used anyway.

    I talked to dozen of ISP type customers who have mentioned this to me.
    Probably over the last 8 years.

    I am glad to hear you think providers offer managerment VPN services. And
    I agree beacuse my data points validate that too.

>> Now you've lost me... What sort of CE-based VPN do you know of that doesn't require CE-PE encryption? If there isn't any encryption and the traffic is being carried over a shared infrastructure then the provider somehow needs to provide VPN isolation, which requires partitioning of resources at the edge (PE), i.e. a PE based VPN is required.
 
    I am talking about PE-based VPNs don't provide encrpytion in and of 
    itself.

>> What's the fundamental problem then?

    The architecture needs to be more distributed so it can scale better.

    It simply comes down to where in the network you put the functionality.

>> And?... So what's your point? If I take the back seats out of my car to improve the power-to-weight ratio then it might go a few miles an hour faster, but it won't be quite as useful. Anything can be optimised, but it still needs to support key functions, and I think VPN support is quite important to the majority of providers.

    To follow your analogy, definitely keep the back seats in, but take a
    look at what you are carrying in the trunk. Maybe you should move it
    to your wife's SUV because it is more appropriate to be there.

Dino

_______________________________________________
pim mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pim

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 20 Oct 2005 22:28:29 -0700

Cc'ing [email-masked] for continuity.

>> e.g. given a VPN with a few hundred sites, CE to CE based routing 
>> exchange is non-trivial as can be proven by looking at any SP network 
>> that has that number of edge boxes. The policy-based edge features on a 
>> PE allow a SP to aggregate the control-plane problem as well as traffic. 
>> Contrary to your claim, it makes the service more scalable from a 
>> management point of view.

    It doesn't make it more scalable it makes it more centralized. And
    distributing the problem doesn't make management harder. There are
    ways to do auto-discovery of CEs for a given VPN.    

>> Clearly, you don't want to have to modify the CE configuration when new 
>> CEs are introduced... so a "network discovery" procedure is necessary. I 
>> think we can agree on that one. That is a necessary control plane 
>> interaction.

    You don't have to. There are deployed mechanisms that have already solved
    this.

>>  From a point of routing exchange, e.g. move from a NxCE mesh to some 
>> sort of hierarchical setup (e.g. using BGP route reflection), it also 
>> helps the SP to do this design function once on the core, instead of per 
>> VPN basis...

    No, not once in the core. Once for each new VPN per PE. Plus with a 
    CE-based solution, nothing needs to change in the P or PE routers. They
    simply move bits.

>> Although one has the option to move this routing exchange to the CE 
>> (inter-as option c) very few SPs choose to do so, even on managed 
>> setups. I don't believe they do so out of ignorance.

    Of course they don't choose to do so. There are business reasons to
    lock all sites of a VPN to the same provider. This does not give
    flexibility to their customers. And I'm sure their customers would like
    to have choices. And if they had provider choices, everyone's service
    would improve.

>> Also, i'm sure that you still remember the IP-overlay-on-ATM days and 
>> are quite aware of the issues with overlay networks. The top layer (the 
>> VPN routing in the case) really needs visibility into the lower transit 
>> layers if you want routing to actually follow some reasonably sensible 
>> path on failure.

    Tunneling is overlay networks. VPNs are overlay networks. So don't say
    we don't do it anymore. And what do you call VPLS?

>> The best argument against CE-based VPNs, imho, are the current MVPN 
>> deployements which are VR-based instead of using the same model than 
>> 2547. Having per VPN data groups in the provider core, is undesirable 
>> from an operations an state point of view.

    You don't need per VPN data groups.

>> So contrary to your view, i'd argue that finding ways to aggregate 
>> control and management state in the PE should be one of the top goals 
>> for a VPN solution.

    You are not aggregating, you are doing all of it in one place. Aggregating
    means to *reduce* information, 2547 does not do that. 2547 is a 
    multiplier of routing tables with only memory separation and no *real*
    privacy.

>> No, it doesn't. They just need to be as private as the leased lines they 
>> are replacing. Most entreprises used to run their own FR/ATM network; 
>> VPNs are a tool to outsource that service.

    No, the design should have gone farther. The requirements should have 
    had security (read encryption) built in. 

Dino

-------------------------------------------------------------------------------

X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="222326298:sNHT32368748"
X-BrightmailFiltered: true
X-Brightmail-Tracker: AAAAAA==
X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="128700253:sNHT15953352"
X-BrightmailFiltered: true
X-Brightmail-Tracker: AAAAAA==
X-IronPort-AV: i="3.97,238,1125903600"; 
   d="scan'208"; a="489568432:sNHT22759208"
Date: Thu, 20 Oct 2005 22:05:33 -0700
From: Pedro Roque Marques <[email-masked]>
User-Agent: Mozilla Thunderbird 1.0.7-1.1.fc4 (X11/20050929)
X-Accept-Language: en-us, en
To: Dino Farinacci <[email-masked]>
CC: [email-masked]
Subject: Re: [pim] Fwd: Multicast in MPLS/BGP IP VPNs
In-Reply-To: <[email-masked]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
X-OriginalArrivalTime: 21 Oct 2005 05:05:34.0669 (UTC) FILETIME=[117BB7D0:01C5D5FD]
X-PMX-Version: 4.7.1.128075
X-from-outside-Cisco: 128.107.243.14

Dino Farinacci wrote:

>>> Please can you explain how the CE-based approach is more cost-effective than the PE-based approach? 
>>> You don't need to pay for a "huge honkin PE box" from the outset. PE chassis are relatively cheap and
>>> blades can be added incrementally as customer demand increases. Putting device equipment costs aside,
 >>> how do you think bandwidth costs/efficiency for multicast traffic 
compares using the  CE-based
 >>> approach (where N streams will need to be carried for N receivers 
on the source access link)?
>     multicast) it isn't going to get easier.

Dino,
The "control-plane interactions" are not complexity that PE-based VPNs 
intruduce. The problem exists by itself.

e.g. given a VPN with a few hundred sites, CE to CE based routing 
exchange is non-trivial as can be proven by looking at any SP network 
that has that number of edge boxes. The policy-based edge features on a 
PE allow a SP to aggregate the control-plane problem as well as traffic. 
Contrary to your claim, it makes the service more scalable from a 
management point of view.

Clearly, you don't want to have to modify the CE configuration when new 
CEs are introduced... so a "network discovery" procedure is necessary. I 
think we can agree on that one. That is a necessary control plane 
interaction.

 From a point of routing exchange, e.g. move from a NxCE mesh to some 
sort of hierarchical setup (e.g. using BGP route reflection), it also 
helps the SP to do this design function once on the core, instead of per 
VPN basis...

Although one has the option to move this routing exchange to the CE 
(inter-as option c) very few SPs choose to do so, even on managed 
setups. I don't believe they do so out of ignorance.

Also, i'm sure that you still remember the IP-overlay-on-ATM days and 
are quite aware of the issues with overlay networks. The top layer (the 
VPN routing in the case) really needs visibility into the lower transit 
layers if you want routing to actually follow some reasonably sensible 
path on failure.

The best argument against CE-based VPNs, imho, are the current MVPN 
deployements which are VR-based instead of using the same model than 
2547. Having per VPN data groups in the provider core, is undesirable 
from an operations an state point of view.

So contrary to your view, i'd argue that finding ways to aggregate 
control and management state in the PE should be one of the top goals 
for a VPN solution.

 > encrpyted before I hit the ISP

No, it doesn't. They just need to be as private as the leased lines they 
are replacing. Most entreprises used to run their own FR/ATM network; 
VPNs are a tool to outsource that service.

   Pedro.

-------------------------------------------------------------------------------

---

From: [email-masked]
Date: Thu, 20 Oct 2005 08:21:45 +0100

Dino,

> There are scaling problems with basic PE-based VPNs.

> How do you expect anything that overlays on top of it to work well?

I assume here you are talking about overlaying multicast VPNs over PE-based unicast VPNs? If so, what exactly is it that you believe is overlaid and over what? In the case of IP/GRE PE-based VPN multicast solutions for example, IP/GRE tunnels are used for multicast traffic whereas completely separate MPLS tunnels are used for carrying unicast traffic, i.e. there is no overlay. In addition, depending on the implementation, the multicast memory/processing resources may be completely separate from unicast memory/processing resources. So, exactly what overlay are you talking about?

>  Putting VPN support at the edges (CE-based VPNs) allows them to scale in a
>  distributed manner

> and is more cost-effective because you can avoid the huge honkin PE box cost

> with all sorts of additional issues like lack of privacy, single point of failure, etc...

>     People say that CE-based VPNs are too hard to manage by a service provider.
>     I don't believe it for one minute.

> If cable and DSL providers can put boxes in our homes, we can have script-based or 
> auto-configuring CE-based devices in remote offices. And if and only if you want 
> network based encryption, then you get encrypted packets starting out 
> on the CE-PE link.

>     So let's fix the fundamental problem and not finger point to PIM or any
>     other protocol.

> BTW, we could get BGP to scale better too if it didn't have to support VPNs.

Regards,
Richard

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 24 Oct 2005 10:15:59 -0700

Hi Richard,

Few comments from me .. I am sure Dino will send his points as well ;).

> All solutions/technologies have scaling limits that when reached
> processing/bandwidth resources need to be increased or an alternative
> solution/technology needs to be selected. Please can you elaborate as
> to what the scaling limits are when using PE-based VPNs that cannot
> be overcome by increasing network resources incrementally as the
> number of customers increases, e.g. adding new route reflectors?

Growing number of RRs as proven in many networks including BT's do not 
solve the issue at all simply due to the fact that easy vpnv4 route 
partitioning versus right scheme of per given VPN PE homing is very 
difficult to achieve in practice.

If you have many VPNs which are served by all RRs what have you gained ? 
Do you have at least any examples of real world operators managing VPNv4 
route partitioning among different RR clusters with some success ?

And that is just a tip of the iceberg. Growing number of PEs means full 
mesh of LSPs between them. While with less then 1K it is not an issue 
when you talk about many more PEs it getting you close to cliff effect 
as no one yet measured number of LDP LSPs which are possible to operate, 
no one did real study of the impact to 2457 of network instabilities in 
your customers etc ... (hint: blind dampening is not what some customers 
would expect from a service).

> Really? I think there may be a few operators that will disagree with
> you. Again, please can you explain what the scaling limits are and
> why they can't be overcome by increasing network resources?

Sure everything can be overcome by increasing network resource :) And no 
one will be more happy then vendors ... Maybe few folks just think too 
much out of the box here trying to help those which do not need the help 
:) ?

Or maybe there is a significant group of customer who would like to peer 
to 2-3 cheap DSL providers with few orders lower prices which they would 
otherwise pay for a "2457 VPN service" pick the best path out of their 
sites depending on dynamic measurements, clearly allowing them to 
achieve the same (or better SLAs) which they get today for $$$$ and 
automatically create full and really secure interconnectivy of their 
sites ?

> So?.. Distributed scaling is not unique to the CE based approach, PE
> based VPNs can also scale in a distributed manor. For example, route
> reflectors can be added as the number of customer routes increases
> and new blades can be added to PE chassis incrementally as customer
> demand increases.

You like those RRs don't you :) ? Please describe in detail how do you 
in practice partition your VPNv4 routes ? How your operations team 
consult to which RR provision which PEs provided that such a PEs carry 
more then handful of VPN customers ???

> Please can you explain how the CE-based approach is more
> cost-effective than the PE-based approach? You don't need to pay for
> a "huge honkin PE box" from the outset. PE chassis are relatively
> cheap and blades can be added incrementally as customer demand
> increases. Putting device equipment costs aside, how do you think
> bandwidth costs/efficiency for multicast traffic compares using the
> CE-based approach (where N streams will need to be carried for N
> receivers on the source access link)?

That has been already explained above. Do you charge the same for VPN 
service as one ordinary non business residential customer pays for basic 
DSL to his home ?

What we are saying that this is what the VPN price should be. If that is 
real that is super ! If not this is what we mean by "cost effective". 
The CE routers capable of good IPSec and few BGP session can be 
purchased in Fry's today.

> Now you've lost me... What sort of CE-based VPN do you know of that
> doesn't require CE-PE encryption? If there isn't any encryption and
> the traffic is being carried over a shared infrastructure then the
> provider somehow needs to provide VPN isolation, which requires
> partitioning of resources at the edge (PE), i.e. a PE based VPN is
> required.

The VPNs we have in mind do not have even have a concept of PEs. ;).

Cheers,
R.