Subject: [Idr] Performance-based BGP Routing Mechanism

From: Xuxiaohu <[email-masked]>
Date: Sun, 7 Apr 2013 08:11:17 +0000

Hi all,

Network latency is widely recognized as one of major obstacles in adopting public cloud services (e.g., cloud desktop service) especially in the scenario where the network path between cloud end-users and cloud data centers may traverse multiple Autonomous Systems (ASes). However, the current Border Gateway Protocol (BGP) specification [RFC4271] doesn't take the network latency into account for the best route selection. In other words, the best route selected according to the existing BGP route selection criteria may not be the best from the end-user experience perspective.

How about allowing BGP peers to exchange Network Path Latency as an additional path attribute of the Network Layer Reachability Information (NLRI) and then take it as one of route selection criteria? To avoid BGP route flapping issue caused by network latency frequent change, the network latency could only reflect the line transport latency by neglecting the latency occurred within routers. To make the performance routing paradigm and the vanilla routing paradigm coexist, performance routes could be exchanged between BGP peers as labeled routes by using a specified Subsequent Address Family Identifier (SAFI). These labeled BGP routes with the lowest latency could be selectively provided to those end-users (e.g., cloud desktop end-users) who are sensitive to network latency. 

Any comments?

Best regards,
Xiaohu

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 7 Apr 2013 12:05:52 +0200

Hi Xiaohu,

Few points ..

1. Where in the best path selection algorithm would you propose to add
this comparison step ?

2. Assume normal AS where BGP control plane is not congruent with data
plane. I assume each BGP speakers which changes next hop should add
such accumulated delay to the former next hop .. how would he know if
former next hop is N hops away ? Are you mandating extensions to the
IGP for example in the form of draft-ietf-ospf-te-metric-extensions ?

3. As you know we already have AIGP attribute. Assume you answer on
the above 1 & 2 it seems that the easiest is to generalize such
attribute to carry delays and other values next or instead of IGP
metric. Here one needs to really see if IGP metric is not already
directly proportional to static link propagation delays. If it is your
original problem is solved.

4. Last I think modern and smart applications can get around this
already by self detecting delays between number of servers and choose
the set of most optimal ones they need. For delay sensitive
applications it would be very unwise to continue traditional client to
server 1:1 design model. So perhaps all of the above 1..3 is not
needed :)

Cheers,
R.

> On Sun, Apr 7, 2013 at 10:11 AM, Xuxiaohu <[email-masked]> wrote:
>
> Hi all,
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sun, 7 Apr 2013 21:51:21 +0000

Comments In-Line

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Sunday, April 07, 2013 4:11 AM
To: [email-masked]
Subject: [Idr] Performance-based BGP Routing Mechanism

Hi all,

Network latency is widely recognized as one of major obstacles in adopting public cloud services (e.g., cloud desktop service) especially in the scenario where the network path between cloud end-users and cloud data centers may traverse multiple Autonomous Systems (ASes). However, the current Border Gateway Protocol (BGP) specification [RFC4271] doesn't take the network latency into account for the best route selection. In other words, the best route selected according to the existing BGP route selection criteria may not be the best from the end-user experience perspective.
[Jim U>] Yes.. This was the reason we created the AIGP attribute.. Not really sure if you are assuming that all of the AS domains are under a common admin authority. If not, you need to ensure that the metric is consistent.. 

Any comments?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 8 Apr 2013 03:09:38 +0000

> -----邮件原件-----
> 发件人: UTTARO, JAMES [mailto:[email-masked]]
> 发送时间: 2013年4月8日 5:51
> 收件人: Xuxiaohu; [email-masked]
> 主题: RE: Performance-based BGP Routing Mechanism
> 
> Comments In-Line
> 
> Jim Uttaro
> 
> Network latency is widely recognized as one of major obstacles in adopting
> public cloud services (e.g., cloud desktop service) especially in the scenario
> where the network path between cloud end-users and cloud data centers may
> traverse multiple Autonomous Systems (ASes). However, the current Border
> Gateway Protocol (BGP) specification [RFC4271] doesn't take the network
> latency into account for the best route selection. In other words, the best route
> selected according to the existing BGP route selection criteria may not be the
> best from the end-user experience perspective.
> [Jim U>] Yes.. This was the reason we created the AIGP attribute.. Not really
> sure if you are assuming that all of the AS domains are under a common admin
> authority. If not, you need to ensure that the metric is consistent..

[Xiaohu] Thanks a lot for your advice. Yes, it's safer and easier to deploy such technology across multiple AS domains if these ASes are under a single administrative authority. Otherwise, those ASes need to reach an agreement on this usage. By using BGP capability negotiation mechanism for such performance routing specific SAFI, it may be helpful to leverage the deployment of such technology.

Best regards,
Xiaohu

> How about allowing BGP peers to exchange Network Path Latency as an
> additional path attribute of the Network Layer Reachability Information (NLRI)
> and then take it as one of route selection criteria? To avoid BGP route flapping
> issue caused by network latency frequent change, the network latency could
> only reflect the line transport latency by neglecting the latency occurred within
> routers. To make the performance routing paradigm and the vanilla routing
> paradigm coexist, performance routes could be exchanged between BGP peers
> as labeled routes by using a specified Subsequent Address Family Identifier
> (SAFI). These labeled BGP routes with the lowest latency could be selectively
> provided to those end-users (e.g., cloud desktop end-users) who are sensitive
> to network latency.
> 
> Any comments?
> 
> Best regards,
> Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Mon, 8 Apr 2013 04:07:09 +0000

Hi Robert,

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Robert
> Raszuk
> 发送时间: 2013年4月7日 18:06
> 收件人: Xuxiaohu
> 抄送: [email-masked]
> 主题: Re: [Idr] Performance-based BGP Routing Mechanism
> 
> Hi Xiaohu,
> 
> Few points ..
> 

My preference is to add it just ahead of AS-Path Len comparison step. Of course, the AS-Path attribute could be resorted in advance for loop detection. Anyway, it can be determined once a rogue consensus in the IDR WG is reached.

> IGP for example in the form of draft-ietf-ospf-te-metric-extensions ?

Yes, one possible way is to use the performance-related information provided by the mechanism defined in the above draft. Another possible way is to periodically measure the RTT to BGP NEXTHOPS by some means. In fact, there are additional ways to do that.

> original problem is solved.

If I understand AIGP correctly, there are at least two different points compared to the idea I proposed:

1) the AIGP attribute is used to convey IGP cost. Therefore, the AIGP doc acknowledged the following constrains when using AIGP:

  " If the AIGP attribute is carried across several ASes, each with its
   own IGP domain, it is clear that these procedures are unlikely to
   give a sensible result if the IGPs are different (e.g., some OSPF and
   some IS-IS), or if the meaning of the metrics is different in the
   different IGPs (e.g., if the metric represents bandwidth in some IGP
   domains but represents latency in others). These procedures also are
   unlikely to give a sensible result if the metric assigned to inter-AS
   BGP links (on which no IGP is running) or to static routes is not
   comparable to the IGP metrics.  All such cases are outside the scope
   of the current document." (quoted from section 3.3.3 of draft-ietf-idr-aigp-09)

2) In the AIGP approach, the performance routing paradigm and vanilla routing paradigm are not separated from each other. In contrast, the idea I proposed allows the performance routing paradigm and vanilla routing paradigm to coexist. As such, performance routing could be provided as a value-added service only to those users who are sensitive to network latency while still providing vanilla routing to other users who are not care about network latency. Anyway, IMHO, AIGP approach and my proposal are targeting for different scenarios.

> server 1:1 design model. So perhaps all of the above 1..3 is not

Regarding this point, I want to hear opinions from co-authors of AIGP draft and draft-ietf-ospf-te-metric-extensions as well :)

Best regards,
Xiaohu

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 8 Apr 2013 18:47:18 +0000

Not sure I understand.. You state

" How about allowing BGP peers to exchange Network Path Latency as an
> and then take it as one of route selection criteria?"

Doesn't this imply that each AS must use a consistent approach to evaluating the latency information being carried? 

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Xuxiaohu
Sent: Sunday, April 07, 2013 11:10 PM
To: UTTARO, JAMES; [email-masked]
Subject: Re: [Idr] Performance-based BGP Routing Mechanism

Best regards,
Xiaohu

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 8 Apr 2013 15:57:57 -0400

On Mon, Apr 8, 2013 at 2:47 PM, UTTARO, JAMES <[email-masked]> wrote:

> Doesn't this imply that each AS must use a consistent approach to
> evaluating the latency information being carried?
>

clearly this will work out well in the real world...

---

From: "Dickson, Brian" <[email-masked]>
Date: Mon, 8 Apr 2013 20:00:12 +0000

While latency may be an obstacle to cloud desktop services, BGP is very
likely not the place to address that.

Consider what you are trying to achieve - are you trying to use the same
IP subnet in multiple places ("anycast"), or do you want each location to
be independently operated and addressed?

Latency is probably much better used as a discriminator, at a higher level
than BGP. For instance, "global server load balancers" are pretty much
designed for this use case, and can operate independent of what is behind
the GSLB, number of locations, etc.

Or, client-server applications dedicated to the "cloud desktop" instance,
may be better suited to handling all the specific needs, especially for
the desktop-specific particulars.

Otherwise, you are likely to end up with "yet another VPN over BGP", with
all the baggage that goes with it.

(The main goal of BGP is interoperability, and multiple non-interoperable
VPN frameworks sitting on top of BGP are the antithesis of exactly that
interoperability.)

But, that is my opinion only.

Brian

On 4/7/13 4:11 AM, "Xuxiaohu" <[email-masked]> wrote:

>Hi all,
>
>Network latency is widely recognized as one of major obstacles in
>adopting public cloud services (e.g., cloud desktop service) especially
>in the scenario where the network path between cloud end-users and cloud
>data centers may traverse multiple Autonomous Systems (ASes). However,
>the current Border Gateway Protocol (BGP) specification [RFC4271] doesn't
>take the network latency into account for the best route selection. In
>other words, the best route selected according to the existing BGP route
>selection criteria may not be the best from the end-user experience
>perspective.
>
>additional path attribute of the Network Layer Reachability Information
>(NLRI) and then take it as one of route selection criteria? To avoid BGP
>route flapping issue caused by network latency frequent change, the
>network latency could only reflect the line transport latency by
>neglecting the latency occurred within routers. To make the performance
>routing paradigm and the vanilla routing paradigm coexist, performance
>routes could be exchanged between BGP peers as labeled routes by using a
>specified Subsequent Address Family Identifier (SAFI). These labeled BGP
>routes with the lowest latency could be selectively provided to those
>end-users (e.g., cloud desktop end-users) who are sensitive to network
>latency. 
>
>Any comments?
>
>Best regards,
>Xiaohu

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 8 Apr 2013 21:21:41 +0000

If there are multiple AS domains under one over-arching authority or a model where there may be many AS domains riding atop the same IGP and services span these AS domains a consistent approach to IGP cost metric is required.. I am not sure of the use case described by Xuxiaohu..

Jim Uttaro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Christopher Morrow
Sent: Monday, April 08, 2013 3:58 PM
To: UTTARO, JAMES
Cc: Xuxiaohu; [email-masked]
Subject: Re: [Idr] Performance-based BGP Routing Mechanism

On Mon, Apr 8, 2013 at 2:47 PM, UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> wrote:
Not sure I understand.. You state

---

From: Ilya Varlashkin <[email-masked]>
Date: Mon, 8 Apr 2013 23:49:33 +0200

Hi,

besides what others have already said, there are some more things to take into account.

First, if all involved AS's are under single administration we can resort to MPLS-TE (including offline tunnel management). From this perspective latency-based routing becomes unnecessary because we can do much more with TE.

Second, if one or more AS's are NOT under single administration, then latency information in BGP becomes irrelevant because chances are nobody is going to trust it. Not only somebody can advertise junk values, but if somebody uses per-class MPLS-TE forwarding then there is no guarantee that advertised BGP latency is the same for all traffic.

Third, the problem you're trying to solve is essentially "to build latency-based path between end-points (client-server)". This seems more work for something like RSVP or call-admission-control and not for a routing protocol.

Cheers,
iLya

> How about allowing BGP peers to exchange Network Path Latency as an additional path attribute of the Network Layer Reachability Information (NLRI) and then take it as one of route selection criteri…

/iLya

---

From: Randy Bush <[email-masked]>
Date: Tue, 09 Apr 2013 08:22:48 +0900

> evaluating the latency information being carried?

that would be easy.  but what it implies is that ALL ASs must have the
SAME consistent metric space, which is what i think you meant.  and cash
should fall from the sky.

randy

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 9 Apr 2013 06:52:02 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Ilya
> Varlashkin
> 发送时间: 2013年4月9日 5:50
> 收件人: Xuxiaohu
> 抄送: [email-masked]
> First, if all involved AS's are under single administration we can resort to
> MPLS-TE (including offline tunnel management). From this perspective
> latency-based routing becomes unnecessary because we can do much more
> with TE.

Yes, inter-AS TE is an alternative way. However, the scalability of inter-AS TE may become a big concern once the amount of TE LSPs is very large.

> Second, if one or more AS's are NOT under single administration, then latency
> information in BGP becomes irrelevant because chances are nobody is going to
> trust it. Not only somebody can advertise junk values, but if somebody uses
> per-class MPLS-TE forwarding then there is no guarantee that advertised BGP
> latency is the same for all traffic.

Agree that these issues need further considerations.

> Third, the problem you're trying to solve is essentially "to build latency-based
> path between end-points (client-server)". This seems more work for something
> like RSVP or call-admission-control and not for a routing protocol.

AFAIK, end-to-end RSVP is not practical anymore in today's large-scale networks. 

Best regards,
Xiaohu

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Tue, 9 Apr 2013 12:05:10 +0200

> > First, if all involved AS's are under single administration we can
> > resort to MPLS-TE (including offline tunnel management). From this
> > perspective latency-based routing becomes unnecessary because we can
> > do much more with TE.
> 
> Yes, inter-AS TE is an alternative way. However, the scalability of
inter-AS TE
> may become a big concern once the amount of TE LSPs is very large.
> 

In our (multi-AS) network latency-based routing is already built into
current network architecture (through IGP cost and BGP MED). My feeling is
that this is also the case for most other networks (possibly also using
AIGP). It's rather not trivial to imagine a case where (IGP, MED, AIGP)
would not suffice. And where it would not suffice quite likely only few TE
tunnels could complete the solution. Or do you have a practical (rather than
theoretical) case where this would not hold?
 
> > Second, if one or more AS's are NOT under single administration, then
> > latency information in BGP becomes irrelevant because chances are
> > nobody is going to trust it. Not only somebody can advertise junk
> > values, but if somebody uses per-class MPLS-TE forwarding then there
> > is no guarantee that advertised BGP latency is the same for all traffic.
> 

It will get hairy for sure. But is the problem actually there and worth the
effort?

> > Third, the problem you're trying to solve is essentially "to build
> > latency-based path between end-points (client-server)". This seems
> > more work for something like RSVP or call-admission-control and not for
a
> routing protocol.

That was just an example, and RSVP was mentioned to emphasize the issue -
today nobody lets foreign traffic to request resources, so why would that be
different with latency-based routing?

Kind regards,
iLya

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 9 Apr 2013 11:27:24 +0000

> 发送时间: 2013年4月9日 18:05

It's great to hear that you are actually using that approach in your production network . In fact, we had ever implemented that feature in our router product before by modifying the current BGP standard behaviors as follows: 1) allow the MED to carry the IGP cost which is set to latency metric; 2) allow the MED value to be cumulated; 3) modify the route selection process to compare the MED anyway, no matter the received routes are from the same neighboring AS, and no matter the routes are IBGP or EBGP routes; 4) perform the MED ahead of LOCAL_PREF and AS_PATH length. Unfortunately, feedbacks from a few of our customers show that this modification destroys the vanilla routing paradigm (especially collide with the current MED standard usage as shown below) which is still useful for them. That's why I proposed here to keep the performance-based routing paradigm and vanilla routing paradigm to be able to coexist by using separate SAFI codes.

        " MULTI_EXIT_DISC is only comparable
         between routes learned from the same neighboring AS (the
         neighboring AS is determined from the AS_PATH attribute)".------- Quoted from RFC4271

By the way, the AIGP draft has also mentioned the drawbacks of using MED to convey latency as follows:

   "There are in fact some implementations which already do something
   like this, using the MULTI_EXIT_DISC (MED) attribute to carry the IGP
   metric.  However, that doesn't really provide IGP-like "shortest
   path" routing, as the BGP decision process gives priority to other
   factors, such as LOCAL_PREF and AS_PATH length.  Also, the standard
   procedures for use of the MED do not ensure that the IGP metric is
   properly accumulated so that it covers all the links along the path"

Xiaohu

---

From: Eric Rosen <[email-masked]>
Date: Tue, 09 Apr 2013 10:52:53 -0400

> public cloud services (e.g., cloud desktop service) especially in the
> scenario where the network path between cloud end-users and cloud data
> centers may traverse multiple Autonomous Systems (ASes).

As others have pointed out, using the "least latency" network path does not
necessarily remove network latency as an obstacle to adopting this sort of
application. 

But for cases where least latency routing through a set of ASes is desired,
AIGP would seem to work perfectly well as long as the metric is agreed.
(Though such agreement, as others have pointed out, is highly unlikely among
an arbitrary set of SPs.)  

> the AIGP attribute is used to convey IGP cost

It could be used to convey any other metric that is agreed upon.

> To make the performance routing paradigm and the vanilla routing paradigm
> coexist, performance routes could be exchanged between BGP peers as
> labeled routes by using a specified Subsequent Address Family Identifier
> (SAFI).

If you did something like this, you could attach AIGP to the "performance
routes" and not to the "vanilla routes".  However, ISPs all around the world
are unlikely to increase their routing load so that some particular ISP's
customers can reduce their latencies a little bit.

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 10 Apr 2013 02:20:25 +0000

> -----邮件原件-----
> 发件人: [email-masked] [mailto:[email-masked]] 代表 Eric Rosen
> 发送时间: 2013年4月9日 22:53
> 收件人: Xuxiaohu
> 抄送: [email-masked]; [email-masked]

Hi Eric,

Thanks a lot for your valuable comments in advance.

Akamai Sureroute (http://www.akamai.com/dl/feature_sheets/fs_edgesuite_sureroute.pdf) is a good example that performance-based routing has its buyers. However, Akamai Sureroute is an overlay based approach, and therefore those alternative paths between Akamai edge
servers selected based upon Sureroute is still not optimal enough from performance perspective since it could not influence the path selection between any pair of its edge servers, unless the density of its edge servers is large enough, e.g., each physical router in the network is attached with an Akamai edge server to an extreme extent. Why couldn't ISPs who have the full control of their physical network resources do something like what Akamai Sureroute has done, even only in a limited network scope or only for their key customers? By doing so, it would not increase their routing load significantly since there is no need to execute performance-based routing for all BGP prefixes, but only for a limited number of prefixes (e.g., prefixes of those customers who have paid for such value-added service).

Any further comments are welcome.

Best regards,
Xiaohu

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 10 Apr 2013 06:20:25 +0200

Hi Xuxiaohu,

> Why couldn't ISPs who have the full control of their physical network
> resources do something like what Akamai Sureroute has done, even
> only in a limited network scope or only for their key customers?

How does it help to do that on a "limited network scope" ? Customers
are even local to ISP (single AS or multi ASes - where AIGP can be
used) and then ISP can do whatever they like to please their customers
or customers applications cross inter-provider boundary and then you
are our of luck.

So I guess what you are hearing is that doing it on "limited network
scope" is as good as not doing it at all.

Again Akamai has it's own overlay they use .. completely different
service then hops of ISPs which often fight for the same customers.

Rgs,
R.

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 12 Apr 2013 09:24:08 +0000

> -----邮件原件-----
> 发件人: Xuxiaohu
> 发送时间: 2013年4月9日 14:52
> 收件人: 'Ilya Varlashkin'
> 抄送: [email-masked]
> 主题: re: [Idr] Performance-based BGP Routing Mechanism
> 
> 
> 
> > -----邮件原件-----
> > besides what others have already said, there are some more things to take
> into
> > account.
> >
> Yes, inter-AS TE is an alternative way. However, the scalability of inter-AS TE

As an alternative to inter-AS TE, one could use BGP-over-TE LSP tunnels (i.e., BGP-signaled LSP over RSVP-TE LSP) in a much scalable way. One application example of such LSP tunnel is inter-AS MPLS-based VPN solution option C (i.e., VPN Label over BGP Label over TE Label). 

As such, when deploying this performance-based BGP routing mechanism across multiple AS’s which are within a single administrative domain, it’s RECOMMENTED to deliver a packet from a BGP speaker to the BGP NEXT_HOP with each AS via tunnels, especially TE LSP tunnels. Furthermore, it’s strongly RECOMMENDED to use the latency metric carried in Unidirectional Link Delay Sub-TLV [OSPF-TE-EXT] [ISIS-TE-EXT] if possible, rather than the TE metric [RFC3630] [RFC5305] or IGP metric to perform the C-SPF calculation, unless the TE metric or the IGP metric has already been set to the link latency metric. In this way, it would avoid the need for timely measurement of end-to-end latency between IBGP peers by some means. Furthermore, this cumulated link latency is more stable than the measured network latency since the former could exclude the factor of internal latency occurred within those transit routers along the forwarding path between BGP peers.

As such, one can use a BGP-over-TE LSP tunnel with the lowest end-to-end latency for inter-AS MPLS-based VPN option C. Besides, by together using either BGP Tunnel-Endpoint Extended Community (http://tools.ietf.org/html/draft-xu-softwire-tunnel-endpoint-03) or BGP Remote-Next-Hop Path Attribute (http://tools.ietf.org/html/draft-vandevelde-idr-remote-next-hop-03), this lowest-latency BGP-over-TE LSP tunnel could be easily used for conveying the latency-sensitive internet traffic across multiple AS's as well.

Best regards,
Xiaohu