Subject: updated draft-gs-l3vpn-scaling & agenda request

From: "George, Wes" <[email-masked]>
Date: Tue, 6 Mar 2012 16:27:17 -0500

Hello all -

Rob and I have completed a revision of our draft discussing L3VPN scaling considerations. We've made some changes to the document structure to make it flow better, and we think that we've added enough of the body that it is ready for discussion during a WG meeting. However, since L3VPN is not meeting during IETF in Paris, we're wondering if we should perhaps ask for time in the Routing area open meeting/RTGAREA WG instead?

Either way, comments are still very welcome, especially if you can help us bolster the currently weak section on multicast VPN scale.

Abstract

   This document discusses scaling considerations unique to
   implementation of Layer 3 (IP) Virtual Private Networks, discusses a
   few best practices, and identifies gaps in the current tools and
   techniques which are making it more difficult for operators to cost-
   effectively scale and manage their L3VPN deployments.

http://tools.ietf.org/html/draft-gs-vpn-scaling-01

Thanks,

Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: <[email-masked]>
Date: Thu, 8 Mar 2012 14:16:12 +0100

Hi George,

Here is my feedback from your document :

1) PE-CE BCP :

I pretty agree with the content of the section.
But I would like to add some stuffs, maybe can we do some recommandation or at least a constation between usage of static/IGP/BGP.
I see this in this way :
- Some years ago, static routing couldn't provide dynamicity of failure detection for multiconnected site where PE-CE link layer not able to detect failure (ethernet non direct link ...), but today usage of BFD with static routing ensure detection in all cases, so static routing can be used for multiconnected sites.

- IGP : running IGP between PE-CE, IMHO, has a sense only to extend customer IGP between sites and make SP network transparent. 

- BGP : BGP is a well designed protocol that can be used for two purposes :
	- providing dynamic advertisement of lot of routes (it seems impossible to provision hundreds of static routes for an access !)
	- providing failure detection

I took IGP case a bit out of my analysis ...
For a monoconnected site, we could see that static routing is fine where number of routes is low (threshold to define : 10/15 ?) and where routes are not changing everyday (otherwise provision activity on PE would be important for this access), otherwise BGP would be prefered. In this case, there is no need of fast failure detection, as it's a monoconnected site. So, setting minimum holdtime to a high value (180) to protect PE may be fine (no need of BFD). 

For a multiconnected site, whatever the protocol used, as you mentionned, it's better to rely on BFD or other connectivity detection mechanism for fast detection rather than tuning protocol timers.The main issue is that CPE are sometimes low cost and not supporting BFD :(
Then choosing protocol is just a matter of number/dynamicity of routes on the access (same as for monoconnected sites) -> choose static when number of route low, BGP otherwise. In case of BGP, if BFD is used, setting minimum holdtime to a high value (180) to protect PE may be fine (BFD ensuring detection).  If BFD not available on CPE, for this specific session, setting a minimum holdtime to a protecting/tested value (15) to protect PE and as you mention number of session with fast timers must be tracked.

Compared to what you mention, the main point to be added in BCP is the difference that should be made between monoconnected sites (no BFD , no fast detection needed) and multiconnected sites (fast detection needed). Sometimes, some SP are using same rule for all BGP access, but it's not pretty good for scaling ...

One other issue to deal with could be persistent flapping of PE-CE BGP session (link issue ... Negotiation issue ...) => BGP DampPeerOscillation needed there but not well implemented

2) Network Event

As for PE-CE issues, we can fall into process priorization issue there. When a PE loose a direct link, or when there is a link failure near a set of PEs. PEs have to update their FIB and possibly hundreds thousands of VPN routes because the change cause an interface change or a MPLS transport label change. We already saw that some routers with bad (old) FIB implementations and bad process priorization are going at 100% CPU updating ISIS routes in RT and FIB and then VPN routes in RT and FIBs. During the 100% CPU, router is loosing PE-CE BGP sessions or ISIS adjacency because of bad process priorization.
Now with H-FIB implementation in codes, the issue is more hidden as it requires less processing :)

3) Route Scale

In the number of routes a PE must support, you can add ISIS routes, LDP FECs, and possibly TE tunnels that are impacting the global scaling too.

Here I propose to separate this section between PE scaling, ASBR scaling, RR scaling and address each part separately. You are mainly talking about PEs here but ASBR and RRs are bottlenecks too.

"Most PE routers use the absence of a
   given VRF instance (or RD/RT filtering) to limit the number of routes
   that they must actually carry, but this is sometimes of limited
   utility for a couple of reasons.  First, it leads to an inconsistent
   routing table footprint from one PE router to the next, and it can
   change with every new customer turned up on the router.  This leads
   to non-deterministic performance and scale."

=> I don't agree on all stuffs there. Limiting routes imported by PE is clearly helping controlplane. If you are importing all routes, then scaling impact is clearly implementation dependant : at least, router will have more memory consumption (so it should support millions of BGP routes !), and possibly CPU usage too (more nexthop reachability computation depending on how it's done...).
As you mentionned, if PE doesn't import all routes, it requires to send route-refresh to RR each time a new VRF is provisionned. With million routes on RR, it's impacting, as it could take some minutes (5-10-20 min) to receive the routes , and only few routes will be accepted , all others will be denied. Formatting RIB-OUT upon route-refresh is something costly for the RR (could impact transient update propagation time). In our case, we are aggregating route-refresh request at RR level every x seconds to permit the RR to format one time and serve multiple PEs with the same update formatting action.

RTC is clearly helping there by just formatting/sending the requested routes and I think there is no issue with using RTC (this is another debate !)

"First, it leads to an inconsistent
   routing table footprint from one PE router to the next, and it can
   change with every new customer turned up on the router" 
=> Yes, but is this an issue ? I agree that it could be better that all PEs having same set of routes, but even with current hardware , this doesn't scale ...

"This leads
   to non-deterministic performance and scale." =>  VPN footprint of customers could be really different, I agree that some customers are spanning among most of PEs, but it's not the case of all. Based on our experience, PEs have different profiles in terms of VRFs.

"In addition, customers may request the use
   of BGP multipath for faster failover or better load balancing, which
   has the net effect of installing more active routes into the table,
   rather than simply selecting the single best path."
=> I think MP has a greater impact on FIB rather than in pure controlplane even if there is generally FIB structures in controlpane ...

Regarding RD policy, I agree with your points, but now the choice can change as there is some solution like add path, ORR, best external that could permit fast restoration with same RD policy (as in non VPN environment) => but I agree that you still do not increase number of nets, but you will increase number of paths ...

Regards,

Stephane

-----Message d'origine-----
De : [email-masked] [mailto:[email-masked]] De la part de George, Wes
Envoyé : mardi 6 mars 2012 22:27
À : [email-masked]
Cc : [email-masked]; [email-masked]
Objet : updated draft-gs-l3vpn-scaling & agenda request

Hello all -

Abstract

Thanks,

Wes George

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 8 Mar 2012 10:50:44 -0600

> => Yes, but is this an issue ? I agree that it could be better that all PEs having
> same set of routes, but even with current hardware , this doesn't scale ...

+1. Frankly, why load the router with routes that it doesn't need (because it has no customers).
Inconsistency in VPN routing table per PE is desired for sanity, IMO.

>    rather than simply selecting the single best path."

True, but that's independent of L3VPN.  

Cheers,
Rajiv

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of [email-masked]
> Sent: Thursday, March 08, 2012 8:16 AM
> To: George, Wes; [email-masked]
> Subject: RE: updated draft-gs-l3vpn-scaling & agenda request
> 
> 
>  Hi George,
> 
> 
> But I would like to add some stuffs, maybe can we do some recommandation
> or at least a constation between usage of static/IGP/BGP.
> I see this in this way :
> - Some years ago, static routing couldn't provide dynamicity of failure
> detection for multiconnected site where PE-CE link layer not able to detect
> failure (ethernet non direct link ...), but today usage of BFD with static
> routing ensure detection in all cases, so static routing can be used for
> multiconnected sites.
> 
> - IGP : running IGP between PE-CE, IMHO, has a sense only to extend
> customer IGP between sites and make SP network transparent.
> 
> 	- providing dynamic advertisement of lot of routes (it seems
> impossible to provision hundreds of static routes for an access !)
> For a monoconnected site, we could see that static routing is fine where
> number of routes is low (threshold to define : 10/15 ?) and where routes are
> not changing everyday (otherwise provision activity on PE would be
> important for this access), otherwise BGP would be prefered. In this case,
> there is no need of fast failure detection, as it's a monoconnected site. So,
> setting minimum holdtime to a high value (180) to protect PE may be fine (no
> need of BFD).
> 
> For a multiconnected site, whatever the protocol used, as you mentionned,
> it's better to rely on BFD or other connectivity detection mechanism for fast
> detection rather than tuning protocol timers.The main issue is that CPE are
> sometimes low cost and not supporting BFD :(
> Then choosing protocol is just a matter of number/dynamicity of routes on
> the access (same as for monoconnected sites) -> choose static when number
> of route low, BGP otherwise. In case of BGP, if BFD is used, setting minimum
> holdtime to a high value (180) to protect PE may be fine (BFD ensuring
> detection).  If BFD not available on CPE, for this specific session, setting a
> minimum holdtime to a protecting/tested value (15) to protect PE and as you
> mention number of session with fast timers must be tracked.
> 
> Compared to what you mention, the main point to be added in BCP is the
> difference that should be made between monoconnected sites (no BFD , no
> fast detection needed) and multiconnected sites (fast detection needed).
> Sometimes, some SP are using same rule for all BGP access, but it's not pretty
> good for scaling ...
> 
> One other issue to deal with could be persistent flapping of PE-CE BGP
> session (link issue ... Negotiation issue ...) => BGP DampPeerOscillation
> needed there but not well implemented
> 
> 
> 2) Network Event
> 
> As for PE-CE issues, we can fall into process priorization issue there. When a
> PE loose a direct link, or when there is a link failure near a set of PEs. PEs have
> to update their FIB and possibly hundreds thousands of VPN routes because
> the change cause an interface change or a MPLS transport label change. We
> already saw that some routers with bad (old) FIB implementations and bad
> process priorization are going at 100% CPU updating ISIS routes in RT and FIB
> and then VPN routes in RT and FIBs. During the 100% CPU, router is loosing
> PE-CE BGP sessions or ISIS adjacency because of bad process priorization.
> In the number of routes a PE must support, you can add ISIS routes, LDP
> FECs, and possibly TE tunnels that are impacting the global scaling too.
> 
> Here I propose to separate this section between PE scaling, ASBR scaling, RR
> scaling and address each part separately. You are mainly talking about PEs
> here but ASBR and RRs are bottlenecks too.
> 
> => I don't agree on all stuffs there. Limiting routes imported by PE is clearly
> helping controlplane. If you are importing all routes, then scaling impact is
> clearly implementation dependant : at least, router will have more memory
> consumption (so it should support millions of BGP routes !), and possibly CPU
> usage too (more nexthop reachability computation depending on how it's
> done...).
> As you mentionned, if PE doesn't import all routes, it requires to send route-
> refresh to RR each time a new VRF is provisionned. With million routes on RR,
> it's impacting, as it could take some minutes (5-10-20 min) to receive the
> routes , and only few routes will be accepted , all others will be denied.
> Formatting RIB-OUT upon route-refresh is something costly for the RR (could
> impact transient update propagation time). In our case, we are aggregating
> route-refresh request at RR level every x seconds to permit the RR to format
> one time and serve multiple PEs with the same update formatting action.
> 
> RTC is clearly helping there by just formatting/sending the requested routes
> and I think there is no issue with using RTC (this is another debate !)
> 
> => Yes, but is this an issue ? I agree that it could be better that all PEs having
> same set of routes, but even with current hardware , this doesn't scale ...
> 
> "This leads
>    to non-deterministic performance and scale." =>  VPN footprint of
> customers could be really different, I agree that some customers are
> spanning among most of PEs, but it's not the case of all. Based on our
> experience, PEs have different profiles in terms of VRFs.
> 
> 
> => I think MP has a greater impact on FIB rather than in pure controlplane
> even if there is generally FIB structures in controlpane ...
> 
> 
> Regarding RD policy, I agree with your points, but now the choice can change
> as there is some solution like add path, ORR, best external that could permit
> fast restoration with same RD policy (as in non VPN environment) => but I
> agree that you still do not increase number of nets, but you will increase
> number of paths ...
> 
> 
> Regards,
> 
> Stephane
> 
> 
> 
> 
> 
> 
> 
> 
> Rob and I have completed a revision of our draft discussing L3VPN scaling
> considerations. We've made some changes to the document structure to
> make it flow better, and we think that we've added enough of the body that
> it is ready for discussion during a WG meeting. However, since L3VPN is not
> meeting during IETF in Paris, we're wondering if we should perhaps ask for
> time in the Routing area open meeting/RTGAREA WG instead?
> 
> Either way, comments are still very welcome, especially if you can help us
> bolster the currently weak section on multicast VPN scale.
> 
> Abstract
> 
> This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or subject to
> copyright belonging to Time Warner Cable. This E-mail is intended solely for
> the use of the individual or entity to which it is addressed. If you are not the
> intended recipient of this E-mail, you are hereby notified that any
> dissemination, distribution, copying, or action taken in relation to the
> contents of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify the sender
> immediately and permanently delete the original and any copy of this E-mail
> and any printout.
> 
> __________________________________________________________
> __________________________________________________________
> _____
> 
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> As emails may be altered, France Telecom - Orange is not liable for messages
> that have been modified, changed or falsified.
> Thank you.

---

From: "George, Wes" <[email-masked]>
Date: Thu, 8 Mar 2012 16:12:15 -0500

Thanks for the comments, we'll certainly incorporate them into our next revision. A couple of responses inline below...

Wes

> From: [email-masked] [mailto:[email-masked]]

> => I don't agree on all stuffs there. Limiting routes imported by PE is
> clearly helping controlplane. If you are importing all routes, then scaling
> impact is clearly implementation dependant : at least, router will have more
> memory consumption (so it should support millions of BGP routes !), and
> possibly CPU usage too (more nexthop reachability computation depending on how
> it's done...).

[WEG] I think I need to clarify what we meant here - this is not saying that limiting which routes the PE imports based on which VRFs are configured is a bad thing. Rather, it is pointing out that this doesn't always help very much when considering how to increase scale. It it only helps if the same VRFs aren't present on every router. If the VRFs carrying the most routes are present on every PE in the network, they're being imported and carried in every PE, meaning you gain less value from the optimization. Yes, it's true that not every VRF is present on every router, but it often makes more of a difference whether the VRF has a large or small number of routes, and the provider has little control over it - in other words, it's a gamble.
The comment about inconsistent routes from one router to the next is more about managing capacity and provisioning - as noted other places in the document, this is about trying not to strand capacity. This inconsistency means that it's one more scaling vector that must be tracked when determining the appropriate place to provision the next customer.

> => Yes, but is this an issue ? I agree that it could be better that all PEs
> having same set of routes, but even with current hardware , this doesn't scale
> ...
[WEG] Again, not saying that you should do the opposite, only pointing out the result

---

From: "George, Wes" <[email-masked]>
Date: Thu, 8 Mar 2012 16:19:07 -0500

> From: Rajiv Asati (rajiva) [mailto:[email-masked]]
> Sent: Thursday, March 08, 2012 11:51 AM
> To: [email-masked]; George, Wes; [email-masked]
> having
[WEG] I think I answered this with my response to Stephane, but if you still have questions after that response, let me know.

> True, but that's independent of L3VPN.
[WEG] That may be so, but my experience has been that SPs are less likely to be willing to enable MP for standard internet traffic, while they will do it for L3VPN. The application as an enterprise network instead of internet connectivity drives a different set of expectations about the performance of the network, design tradeoffs that must be made at the carrier's behest, etc. Keep in mind that in many cases, L3VPNs are replacing old point-to-point networks or L2VPNs using legacy Frame Relay, ATM, or L2TPv3. The customers often don't want to make major architectural changes to their routing, and therefore expect the SP to do the same things that they were doing between their routers before. This may well include multipath. Therefore it's more pertinent in this context.

Thanks
Wes George

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 09 Mar 2012 02:00:19 +0100

Hi,

> [WEG] I think I need to clarify what we meant here - this is not
> saying that limiting which routes the PE imports based on which VRFs
> are configured is a bad thing. Rather, it is pointing out that this
> doesn't always help very much when considering how to increase scale.
> It it only helps if the same VRFs aren't present on every router. If
> the VRFs carrying the most routes are present on every PE in the
> network, they're being imported and carried in every PE, meaning you
> gain less value from the optimization. Yes, it's true that not every
> VRF is present on every router, but it often makes more of a
> difference whether the VRF has a large or small number of routes, and
> the provider has little control over it - in other words, it's a
> gamble. The comment about inconsistent routes from one router to the
> next is more about managing capacity and provisioning - as noted
> other places in the document, this is about trying not to strand
> capacity. This inconsistency means that it's one more scaling vector
> that must be tracked when determining the appropriate place to
> provision the next customer.

Actually this problem you are pointing out can be very well solved by 
using draft-rekhter-l3vpn-virtual-hub idea. Then on any PEs you only 
keep local routes plus pointer to the hub VRF where the actual lookup 
will happen.

If you place such hub PE wisely (note that such hub PE does not need to 
have sites attached to it) for example on the POP to core boundary your 
scaling becomes much more deterministic as compared with today's model.

Further one could enable draft-ietf-grow-simple-va model for virtual hub 
to even further scale data plane when you have large multihomed
sites in a VPN.

Best regards,
R.

---

From: "Daniel Cohn" <[email-masked]>
Date: Tue, 13 Mar 2012 10:30:23 +0200

Hi Wes and Rob,

First of all thanks for writing this draft, it does a great job of
compiling L3VPN scalability issues in one place and I think it will be
very helpful for developers and operators.

Few questions/comments:

- I understand/agree with the general assertion that BGP is inherently
more scalable than IGPs, also in PE-CE application. But I don't
understand the statement that "(IGPs) invoke additional processes on the
router when compared to simply using BGP (which is already going be
running on a router using MP-BGP for VPNs"). This is because even if BGP
is already running on the router as PE-PE protocol, using BGP as PE-CE
protocol will also typically required "additional processes" in the
router as each BGP instance will typically run on a separate BGP
process. Barring specification implementations, I don't see how BGP and
IGPs as PE-CE differ in this aspect.

- In the following paragraph: 

"Where it may be possible to assign a single RD per L3VPN instance, and
hence achieve some level of route aggregation on BGP speakers within the
solution, this has some consequences for both convergence in the VPN
(due to BGP convergence being relied upon) and in its potential to
exacerbate geographic distance between PE and Route-reflector and is
therefore undesirable in some circumstances"

Are you referring to multiple-homed CE scenario, where the same NLRI
will be advertised with different BGP next hops if the same RD is used?
If yes, I think it should be clarified. If not, can you please explain
to which scenario you refer?
Also, what do you mean by "exacerbate geographic distance"? I don't
understand the meaning of this expression nor the problem it describes.

- Regarding the maximum number of routes per VRF limit, I think there
should be some discussion on the PE behavior when the limit is reached,
e.g.: what do you expect PE-CE and PE-PE protocols should do with routes
they receive after the limit has been reached? Should these routes be
stored and installed at the VRF when the route count goes below the
limit (plus some hysteresis threshold of course)? Or should they also be
discarded by the PE-CE/PE-PE routing protocols? If discarded, how will
you resync with the neighbors/peers once the route count goes below the
limit?

Regards,

Daniel

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of George, Wes
Sent: Tuesday, March 06, 2012 11:27 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: updated draft-gs-l3vpn-scaling & agenda request

Hello all -

Rob and I have completed a revision of our draft discussing L3VPN
scaling considerations. We've made some changes to the document
structure to make it flow better, and we think that we've added enough
of the body that it is ready for discussion during a WG meeting.
However, since L3VPN is not meeting during IETF in Paris, we're
wondering if we should perhaps ask for time in the Routing area open
meeting/RTGAREA WG instead?

Either way, comments are still very welcome, especially if you can help
us bolster the currently weak section on multicast VPN scale.

Abstract

Thanks,

Wes George

This E-mail and any of its attachments may contain Time Warner Cable
proprietary information, which is privileged, confidential, or subject
to copyright belonging to Time Warner Cable. This E-mail is intended
solely for the use of the individual or entity to which it is addressed.
If you are not the intended recipient of this E-mail, you are hereby
notified that any dissemination, distribution, copying, or action taken
in relation to the contents of and attachments to this E-mail is
strictly prohibited and may be unlawful. If you have received this
E-mail in error, please notify the sender immediately and permanently
delete the original and any copy of this E-mail and any printout.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 13 Mar 2012 09:57:15 +0100

Hi Daniel,

> IGPs as PE-CE differ in this aspect.

BGP does not have a new process/thread/data structure on a per VPN basis.

All VPN routes are stored in single vpnv4 trie and the only real 
overhead you add is additional cycles for new neighbor(s) update 
generation. To make the routes unique across VPNs RD is part of such 
trie. That is the same on the PE as it on the RR.

Contrary to this when you use IGP on the PE-CE for example OSPF you need 
to keep an istance of LSDB as I am not aware of any efforts to modify 
IGP LSDB to be VPN aware. Same for topology SPF, etc ...

RIP could be implemented much easier to be VPN aware how ever as it's 
use is marginal on the PE-CE I am not sure if this would be the high on 
the vendor's roadmap.

> will be advertised with different BGP next hops if the same RD is used?

Yes you are correct. For multi-homed sites each advertising same prefix 
via different PEs using same RD on both PEs in the corresponding VRFs 
will result in the vpnv4 RR making the best path selection across both 
paths and advertising only a single best path. Then remote PEs would 
receive only one path and could not perform any fast connectivity 
restoration techniques (PIC as example).

> you resync with the neighbors/peers once the route count goes below the

VRF limit is typically reflected by control plane protocols as RIB 
install failure. Those routes therefor are not advertised to CEs/RRs, 
but are still kept in the BGP table.

In those cases the syslog message raises NOC alarm. I think it is 
implementation dependent if they are allowed to be retried to be 
installed into VRFs automagically when the limit allows.

No matter how you put it this is a mess for a given VPN and rather VRF 
limit was designed to be used as red light protecting other VPNs on a 
PE. Perhaps along discussing VRF limit authors should also discuss BGP 
prefix limit as proper correlation of both seems helpful.

Best,
R.

---

From: "Daniel Cohn" <[email-masked]>
Date: Wed, 14 Mar 2012 10:03:58 +0200

Hi Robert, please see inline with [DC].

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]] 
Sent: Tuesday, March 13, 2012 10:57 AM
To: Daniel Cohn
Cc: George, Wes; [email-masked]; [email-masked];
[email-masked]
Subject: Re: updated draft-gs-l3vpn-scaling & agenda request

Hi Daniel,

> IGPs as PE-CE differ in this aspect.

BGP does not have a new process/thread/data structure on a per VPN
basis.

[DC] I think this is getting too much into implementation-specific
issues. What you write above is related to routing information
structure, but this does not necessarily mandate implementation
architecture. For example I am familiar with multi-instance OSPF
implementations that run a single process/thread for all instances,
while internally adding instance information to the LSDB.

[DC] Thanks for the clarification. I suggest you add it to the draft.
You didn't answer the  following question though: " Also, what do you
mean by "exacerbate geographic distance"? I don't understand the meaning
of this expression nor the problem it describes."

[DC] I agree. Also OSPF LS limit.

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 14 Mar 2012 09:16:26 +0100

Hi Daniel,

> architecture.

Very true. I was only bringing examples of most widely known/used 
implementations.

> For example I am familiar with multi-instance OSPF

Correct. But one needs to keep in mind that multi-instance IGP proposals 
for all practical purposes are limited to small amount (4,8,16) while 
L3VPNs clearly goes over 1000s.

> [DC] Thanks for the clarification. I suggest you add it to the draft.

I think Wes will :) I am not co-author of this document.

> of this expression nor the problem it describes."

The way I read this is that it says that using same RD will on one hand 
allow to limit number of paths for a given VPN however among other 
issues it exposes inaccurate BGP best path selection problem. In those 
cases VPNv4/v6 route reflector will be selecting best path in his IGP 
location in the network (or with completely disabled metric to next hop 
comaprison) what will result for some (most) PEs to get suboptimal path 
from such RR.

The problem is well known and some time back I have proposed general 
solution to this issue. It is described in: 
draft-ietf-idr-bgp-optimal-route-reflection-01 and it is applicable to 
both IPv4/IPv6 as well as same RD L3VPNs.

Best regards,
R.

---

From: "George, Wes" <[email-masked]>
Date: Thu, 15 Mar 2012 09:00:37 -0400

Daniel - thanks for the review and the comments.
Robert answered a lot of the questions, so thanks to him for that, a few additional points inline below.

Thanks,

Wes

> From: Daniel Cohn [mailto:[email-masked]]
> Sent: Wednesday, March 14, 2012 4:04 AM
> To: [email-masked]
> Cc: George, Wes; [email-masked]; [email-masked]; [email-masked]
[snip]
> while internally adding instance information to the LSDB.

[WEG] I meant it in a slightly more simplistic way... A standard PE will be running an IGP and BGP for PE-PE and PE-P route exchange. Adding VRFs to BGP will add memory state, but is not usually going to result in spawning completely separate BGP processes. The details are indeed implementation specific, but we thought that this was a reasonable assumption based on the common implementations. Contrast this with a PE that is doing EIGRP, OSPF, etc as PE-CE routing protocol. If IS-IS is the chosen IGP, you're now running another routing protocol when you fire up OSPF. Same is true if you're already running OSPF as your IGP and you fire up EIGRP for a customer. Even if you are running OSPF as IGP and only use OSPF as PE-CE (vs adding EIGRP, etc), you're likely starting up another instance. Yes, it is possible for optimizations that manage it more like BGP does (one instance, but VRF aware to separate things properly within that instance), but IMO that's almost worse, because now you've potentially gone and tied your core IGP convergence and scale limitations to the customer-facing routing machinery. Unless the implementation is intelligent enough to prioritize global table IGP first, and only after that is done proceed with PE-CE, you're now in a race condition, to say nothing about how it handles large networks with lots of elements which can challenge the scaling limitations of the implementation all by themselves. I'm not sure it makes sense to get into this within the document since it is so implementation specific, but I'm open to discussion one way or the other.
> > should be some discussion on the PE behavior when the limit is
[snip]
> prefix limit as proper correlation of both seems helpful.
[WEG] This is an interesting distinction. We likely need to make it clearer in the document, because there are definitely differences in how one might set a max-prefix limit per-session vs how you set the route-limit for the VRF. There are probably some things to add to the gap discussion as well, since there isn't a good method for end-users to know that they have a problem with route-limits. At least with max-prefix exceeded, you get a session shutdown that is a good indicator to both peers that something is amiss. If routes just get dropped, the sending party has no way of knowing why until they call in a trouble ticket and/or the SP notices the alarms on route-limits being exceeded.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 15 Mar 2012 14:11:51 +0100

Hi Wes,

> At least with max-prefix exceeded, you get a session shutdown that is
> a good indicator to both peers that something is amiss. If routes
> just get dropped, the sending party has no way of knowing why until
> they call in a trouble ticket and/or the SP notices the alarms on
> route-limits being exceeded.

I am glad you say so ... especially in the light of new BGP defaults 
where due to treat-as-withdraw upon attribute error in the update 
message sending party has no way of knowing.

Worse until the AS which does this has it's NOC script updated to catch 
it from the syslog even there such holes may get punched completely 
unnoticed.

My hope is that BGP Operational Message may address the issue. In fact I 
think we could update the latest draft to also consider a message when 
vrf-limit is reached.

Best,
R.