Subject: WG Last Call for draft-ietf-l3vpn-virtual-hub

From: Thomas Morin <[email-masked]>
Date: Thu, 04 Oct 2012 11:35:38 +0200

Hello working group,

This email starts a two-week Working Group Last Call on 
draft-ietf-l3vpn-virtual-hub, which is considered mature and ready for a 
working group review.

Please read the document if you haven't read the most recent version, 
and send your comments to the list, no later than Thursday October 18th.

Thank you,

-Thomas & Martin

[ http://tools.ietf.org/html/draft-ietf-l3vpn-virtual-hub ]

---

From: Marco Rodrigues <[email-masked]>
Date: Thu, 4 Oct 2012 14:48:42 +0000

Support. 

Marco.

---

From: "SAAD, SAMIR S" <[email-masked]>
Date: Thu, 4 Oct 2012 15:00:30 +0000

Support

Samir Saad
AT&T Labs

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Morin
Sent: Thursday, October 04, 2012 5:36 AM
To: L3VPN
Subject: WG Last Call for draft-ietf-l3vpn-virtual-hub

Hello working group,

Thank you,

-Thomas & Martin

---

From: Thomas Morin <[email-masked]>
Date: Thu, 04 Oct 2012 17:13:46 +0200

Hello working group,

One precision...

A working group last call is essentially the opportunity to raise 
comments that would not have been made before to be made now, before the 
document moves to IESG, and the opportunity for those who haven't read 
it yet to realize that it is the right time to do so before processes 
get heavier.

Advertising "support" is harmless but less useful than more detailed 
reviews or, for instance, explanations why than the work is considered 
important or statements that the work is mature.

-Thomas

Thomas Morin :
> draft-ietf-l3vpn-virtual-hub, which is considered mature and ready for 
> a working group review.
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 4 Oct 2012 20:18:39 +0200

Hello Thomas,

I think the draft is useful however as communicated to authors of this
draft on 2nd Aug 2011 I am of the opinion that it should be extended
to accomodate even further scaling enhancements not just for SP PEs
but also for actual customer VPN sites.

What I have suggested over a year ago was the extension of this
proposal to CSC case where virtual hub is placed in the service
provider network as given customer's VPN hub and where optimal
forwarding decision can be computed for routing site to site customer
traffic.

That idea is in fact simpler then the one proposed in the current
draft as it does not require import/export RT configuration at the
virtual hub. Instead just like in plane vanilla CSC case each
customer's routes are handled separately and placed into appropriate
virtual hub.

Moreover even going further scaling of such virtual hubs residing in
strategic locations of SP core infrastructure could be accomplished
with simple grid computing or distributed routing platforms being
fully transparent to any need for additional PE's configuration or
what perhaps is more important any per VPN churn in routing
information.

To realize this model non of the currently deployed PEs would need to
be upgraded or even reconfigured provided given SP already provides
CSC service.

Last but not least the data plane outer encapsulation of such virtual
hub could be IP (GRE) therefor opening new opportunities for offering
CSC type of VPNs for much larger customer base then currently limited
to only those sites which are directly attached to given's SP
infrastructure.

Best regards,
R.

On Thu, Oct 4, 2012 at 11:35 AM, Thomas Morin <[email-masked]> wrote:
> Please read the document if you haven't read the most recent version, and
> send your comments to the list, no later than Thursday October 18th.
>
> Thank you,
>
> -Thomas & Martin
>

---

From: <[email-masked]>
Date: Fri, 5 Oct 2012 15:57:08 +0000

Hello Thomas, Martin, 

I have read the draft and believe it's ready for submission to the IESG.

It's a useful proposition to improve the scalability of some PE. In particular in the context where IP/MPLS networks are expanding from the core to the aggregation part of the network, where given both the smaller number of customers using it and the large number of equipments and locations, such PE are more likely to be chosen smaller and/or replaced less often.
Also, the solution can be deployed incrementally with incremental benefit.

Thank you,
Regards,
Bruno

>Sent: Thursday, October 04, 2012 11:36 AM
>To: L3VPN

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: "Luay Jalil" <[email-masked]>
Date: Mon, 8 Oct 2012 00:56:12 -0500

Support (co-author)

This draft is useful in reducing the number of routes on some of the PEs in an MPLS VPN environment

Regards,
Luay Jalil

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Morin
Sent: Thursday, October 04, 2012 4:36 AM
To: L3VPN
Subject: WG Last Call for draft-ietf-l3vpn-virtual-hub

Hello working group,

This email starts a two-week Working Group Last Call on draft-ietf-l3vpn-virtual-hub, which is considered mature and ready for a working group review.

Please read the document if you haven't read the most recent version, and send your comments to the list, no later than Thursday October 18th.

Thank you,

-Thomas & Martin

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 8 Oct 2012 10:38:01 -0700

I agree, the draft improves the efficiency of PE router support for L3
VPNs and is a mature output of the WG.

Cheers,
Andy

---

From: Jon Mitchell <[email-masked]>
Date: Thu, 11 Oct 2012 10:39:04 -0400

Support the solution in this draft and it appears to be a well thought
out solution to allow a SP who does not want to fundamentally change the
service offering (having customer encapsulate/tunnel traffic on the CE,
which eliminates IP based services from being offered by the SP) to
scale a traditional any-to-any MPLS VPN service.

I do have a clarification question though which I probably should have
asked before WGLC:

1.  It appears that changes are not necessarily required on v-spoke
forwarding in all cases.  Maybe the document could be more clear about
what this higher up in the text (referring to last paragraph of section
4 and middle part of section 5, where an option is presented that would
require changes to v-spoke to support the model).  Do the authors intend
that this capability could be deployed for a VPN if some of the v-spokes
are not supporting any behaviors in this draft (defined as "vanilla" in
the Overview, which later is never utilized as a term except to talk
about existing deployments rather than specific PE's)?  Is the only
constraint that they should not be receiving a default route from CE and
that multi-homed sites are provisioned as v-hubs?  If an operator
accepts these restrictions appraoch can a "vanilla" PE be a v-spoke or
did I miss some other requirement?

Thanks,

Jon

On Thu, Oct 04, 2012 at 05:13:46PM +0200, Thomas Morin wrote:
> comments that would not have been made before to be made now, before
> the document moves to IESG, and the opportunity for those who
> haven't read it yet to realize that it is the right time to do so
> before processes get heavier.
> 
> reviews or, for instance, explanations why than the work is
> considered important or statements that the work is mature.
> 
> -Thomas
> 
> 
> Thomas Morin :
> >Hello working group,
> >
> >draft-ietf-l3vpn-virtual-hub, which is considered mature and ready
> >for a working group review.
> >
> >Please read the document if you haven't read the most recent
> >version, and send your comments to the list, no later than
> >Thursday October 18th.
> >
> >Thank you,
> >
> >-Thomas & Martin
> >

---

From: Eric Rosen <[email-masked]>
Date: Fri, 12 Oct 2012 11:02:06 -0400

I have a number of comments on the virtual-hub draft.  Some are minor and/or
editorial, but a number are more substantial.  I think these comments need
to be addressed before the draft is submitted for publication.

I've placed a lot of comments in-line, but let me summarize what I think are
the major issues:

- It is not at all clear what has to be implemented in order to conform to
  this draft.  There is a lot of complexity introduced to handle a few
  specific scenarios, but it's not clear whether it is mandatory to support
  those scenarios.  Is it mandatory to support the scenario where a customer
  gets Internet access via a site that is homed to a V-spoke PE?  Is it
  mandatory to support the scenario where a customer has a site that is
  multihomed to a pair of V-spoke PEs, and the customer is expecting a
  certain load-balancing behavior?

  Much of the discussion about these scenarios is of the sort that says "you
  could do this, you could do that, or you could do some other thing", but
  it's not clear what is required for conformance.  This really needs to be
  clarified. 

- Various procedures seem to assume that a PE, when processing a default
  route, will know whether that route is "the Internet VPN-IP default route"
  or "the non-Internet VPN-IP default route".  Yet there is no clear
  statement of how this is known.  I think it is supposed to be inferred
  somehow from the RTs carried by the route, but the draft never says that a
  VRF has to be provisioned to know that any specific RT is used for this
  particular purpose.

  This is an example of a more general issue: the draft suggests that
  certain routes need to be exported with a different RT (or a different set
  of RTs) than certain other routes, but it doesn't say explicitly that an
  importing VRF has to know that any particular import RT has any particular
  semantics.  This is an area that I think needs to be clarified.

- If V-hub1 receives a packet from the backbone, and the packet is destined
  to a site which is attached both to V-hub1 and to another V-hub, V-hub2,
  V-hub1 is allowed to do load-balancing by choosing whether (a) to send the
  packet directly to the site, or (b) to send the packet to V-hub2.  But I
  don't see anything in the draft that prevents V-hub2 from doing its own
  load-balancing, and perhaps sending the packet back to V-hub1.

- The draft says that a V-hub shouldn't originate a default route if it
  receives a default route from one of its attached CEs.  But it doesn't say
  what to do if the V-hub originates a default route and the CE later sends
  it a default route.  Nor does it say what to do if the CE later withdraws
  the default route.  One can surmise what needs to be done in these cases,
  but the spec should say explicitly what to do.

- In at least one place, the draft states "a PE that acts as a V-spoke of a
  given VPN needs to maintain ... two or more VPN-IP default routes whose
  Next Hops are the addresses of the V-hubs".  This seems to rule out the
  case where the V-spoke and V-hub are separated by an "option B" ASBR.
  (The option B ASBR would change the next hop.)  If this restriction is
  intentional, it should be stated explicitly, and a reason for it should be
  given.

- Hidden in the "fine print" is a significant change to RFC 6514.  This
  draft proposes to allow the setting of the LIR flag in I-PMSI A-D routes,
  while RFC 6514 prohibits it.  Presumably this change is only meant to
  apply when the V-hub/V-spoke architecture is being used, but the draft
  doesn't really say how the receiver of an I-PMSI A-D route is supposed to
  know whether LIR is legal or not, and it doesn't address interoperability
  issues with PEs that adhere to the RFC 6514 rules.  If this change is
  considered desirable, it should really be separated out from this draft
  and proposed in a separate internet-draft.

  I think this should be removed from the V-hub draft.

- The V-hub/V-spoke architecture affects certain multicast scenarios, but not
  others.  In particular, I think it only affects multicast scenarios where
  (a) the source of a multicast flow is at a site attached to a V-spoke, and a
  receiver of that flow is at a site attached to a different V-spoke, or (b)
  the source of a multicast flow is at a site attached to a V-hub, and a
  receiver of that flow is at a site attached to a V-spoke that is not
  associated with that same V-hub.  But it is left as an exercise to the
  reader to figure out what these scenarios are.  The draft should state
  them explicitly.

- In general,when a multicast flow needs to go from one V-spoke to another
  V-spoke, the draft requires the BGP control messages to go from V-spoke to
  V-hub to V-spoke.  I agree that, in order to meet the goals of the draft,
  it is necessary to put the V-hub in the control path.  However, the draft
  also requires the data flow itself to go from V-spoke to V-hub to V-spoke.
  I do not see why the V-hub needs to insert itself into the multicast data
  path.  For unicast, the V-hub needs to be in the data path because it
  needs to do an IP lookup for each data packet.  For multicast, the V-hub
  does not do an IP lookup for each data packet, so I don't see why it needs
  to be in the data path.

  As long as the V-spoke PEs have global table unicast routes to each other,
  it should be possible to keep the V-hubs out of the multicast data path
  between two V-spokes.  (When a V-hub tells a V-spoke which P-tunnel to
  join in order to receive (C-S,C-G) traffic, why can't the V-hub specify a
  P-tunnel that is routed at the PE that is actually attached to the C-S?)

  The draft does not say that the V-spokes don't have global table routes to
  each other, just that they don't have VPN-IP routes to the multicast
  sources.  So I think the draft needs to provide some justification for
  putting the V-hubs in the multicast data path.

- Something should be said about whether this draft is expected to work in a
  "seamless MPLS environment" where there is an ABR between a V-hub and a
  V-spoke.  Even a statement that seamless MPLS (and/or seamless MPLS
  multicast) is out of scope (if that's the authors' intention) would be
  helpful.  If "seamless MPLS multicast" is not out of scope, then something
  needs to be said about the handling of the P2MP Segmented Next Hop
  Extended Community on A-D routes received by the V-hub from V-spokes or
  from other V-hubs.

  (It might also be helpful to say that "MVPN extranet" is out of scope.)

There are also a few more minor issues that I think need to be addressed:  

- I'd like to see some more explicit mention of the asymmetry in the
  procedures.  A V-spoke only imports the default route from certain V-hubs,
  but all V-hubs import all routes from all V-spokes.  I'm not saying that
  this is a problem, just that it is a part of the architectural model and
  should be mentioned explicitly.

- Some may think the following is a very minor issue, but I think it is very
  confusing to use the term "Default I-PMSI A-D route" to refer to a route
  that is not a "default route" and that is also not a route advertising a
  "default I-PMSI".  

- The procedures of the draft do not appear to generalize easily to a
  multi-level hierarchy.  If this is a non-goal, that should be stated
  explicitly. 

More comments appear in-line.

...

                      draft-ietf-l3vpn-virtual-hub-01.txt

...

Abstract

   With BGP/MPLS VPNs any-to-any connectivity among sites of a given
   Virtual Private Network would require each Provider Edge router that
   has one or more of these sites connected to it to hold all the routes
   of that Virtual Private Network. The approach described in this
   document allows to reduce the number of Provider Edge routers that
   have to maintain all these routes by requiring only a subset of these
   routers to maintain all these routes.

Eric> As far as I can tell, the real purpose of this document is enable
Eric> "BGP/MPLS IP VPNs" to provide any-to-any connectivity by using a sort
Eric> of "hub and spoke" overlay.  Only the hubs hold all the routes.
Eric> Spoke-to-spoke traffic is "default routed" from the source spoke to a
Eric> hub, where it is reforwarded to the destination spoke.  I don't think
Eric> the above paragraph captures this very well.

   Furthermore, when Provider Edge routers use ingress replication to
   carry multicast traffic of VPN customers, the approach described in
   this document could allow to reduce bandwidth inefficiency associated
   with ingress replication, and to redistribute the replication load
   among Provider Edge routers.

Eric> I find this quite misleading; whether bandwidth inefficiency is
Eric> reduced or increased depends upon the topology, and upon the location
Eric> of the receivers relative to the location of the source and of the
Eric> hubs.  Also, it's not clear what is meant by "redistribute the
Eric> replication load", or why this is a good thing.

...

2. Overview

   With BGP/MPLS VPNs [rfc4364] providing any-to-any connectivity among
   sites of a given Virtual Private Network (VPN) is usually
   accomplished by requiring each Provider Edge router (PE) that has one
   or more of these sites connected to it to hold all the routes of that
   VPN.  The approach described in this document allows to reduce the
   number of PEs that have to maintain all these routes by requiring
   only a subset of such PEs to maintain all these routes.

   Consider a VPN that has its sites connected to a set of PEs. In the
   context of this VPN we designate a subset of these PEs as "Virtual
   Spoke" PEs (or just Virtual Spokes), while some other (non-
   overlapping) subset of these PEs will be "Virtual Hub" PEs (or just
   Virtual Hubs), with the rest of the PEs in the set will be "vanilla"
   PEs (PEs that implement the [rfc4364] procedures, but do not
   implement procedures specified in this document).

   For the sake of brevity we will use the term "V-hub" to denote a
   Virtual Hub, and "V-spoke" to denote a Virtual Spoke.

   For a given VPN, its set of V-hubs could include not only the PEs
   that have sites of that VPN connected to them, but also PEs that have
   no sites of that VPN connected to them.

Eric> Two paragraphs before this, it says that a Virtual Hub or Spoke (of a
Eric> given VPN) is a subset of the set of PEs that are attached to the sites
Eric> of the VPN.  So how can a V-hub be a PE that has no sites of the VPN
Eric> connected to it?

Eric> Perhaps what is meant is that a particular V-hub VRF, say VRF-1, for a
Eric> given VPN, may import routes from other V-hub and V-spoke VRFs of that
Eric> VPN, even if VRF-1 is not itself connected to a customer site.
Eric> However, this does contradict the second paragraph of this section.

   Note that while in the context of one VPN a given PE may act as a V-
   hub, in the context of another VPN the same PE may act as a V-spoke,
   and vice versa. Thus a given PE may act as a V-hub only for some, but
   not all the VPNs present on that PE. Likewise, a given PE may act as
   a V-spoke only for some, but not all the VPNs present on that PE.

Eric> It might have been better to define "V-hub" and "V-spoke" to be types
Eric> of VRFs, rather than types of PEs.  (Then one wouldn't run into the
Eric> awkwardness of having to say that a given PE is both a V-spoke PE and a
Eric> V-hub PE, depending on which VPN one is speaking of.)

   For a given VPN each V-spoke of that VPN is "associated" with two or
   more V-hubs of that VPN (we use two V-hubs for redundancy to avoid a
   single point of failure). Note that a given V-hub may have no V-
   spokes associated with it.

Eric> Shouldn't it be "one or more"?  Presumably the procedures still work if
Eric> a given V-spoke is associated with only one V-hub.  "Two or more" is
Eric> just a deployment recommendation.

Eric> Consider the set of V-spokes that are associated with V-hub-1.  Is it
Eric> allowable for some V-spokes in the set to be associated with V-hub-2
Eric> (and NOT associated with V-hub-3), while others are associated with
Eric> V-hub-3 (and NOT associated with V-hub-2)?  I think the draft should
Eric> answer this explicitly.

   A PE that acts as a V-hub of a given VPN maintains all the routes of
   that VPN.

Eric> By the way, this statement rules out the possibility of a multi-level
Eric> hierarchy, where a VRF serves as a hub to the lower layer and as a
Eric> spoke to the higher layer.  Is this intentionally ruled out?

   A PE that acts as a V-spoke of a given VPN needs to
   maintain only the routes originated by the sites of that VPN
   connected to that PE,

Eric> suggest "only the routes originated by" --> "only the routes of that
Eric> VPN that are originated by"

   plus two or more VPN-IP default routes whose

Eric> "one or more"?

   Next Hops are the addresses of the V-hubs associated with that V-
   spoke.

Eric> An implication of this is that the V-hubs and associated V-spokes must
Eric> be in the same AS.  Or at least that they are not separated by an ASBR
Eric> with an "option B" interconnect?  Is that the intention?   (I don't
Eric> think it is.)

   This way, only a subset of PEs that have sites of a given VPN,
   namely only the PEs acting as V-hubs of that VPN,  have to maintain
   all the routes of that VPN. PEs acting as V-spokes of that VPN need
   to maintain only a (small) subset of the routes of that VPN.

   When PEs use ingress replication to carry multicast traffic of VPN
   customers, the approach described in this document could allow to
   reduce bandwidth inefficiency associated with ingress replication,
   and to redistribute the replication load among the PEs. This is
   because a PE that acts as a V-spoke of a given VPN would need to
   replicate multicast traffic only to other V-hubs (while other V-hubs
   would replicate this traffic to the V-spokes associated with these V-
   hubs), rather than to all the PEs of that VPN. Likewise, a PE that
   acts as a V-hub of a given VPN would need to replicate multicast
   traffic to other V-hubs, and the V-spokes, but only to the V-spokes
   associated with that V-hub, rather than replicating the traffic to
   all the PEs of that VPN. Limiting replication could be especially
   beneficial if a PE that does the replication has limited replication
   capabilities and/or has links with limited bandwidth.

Eric> I think "if a PE that does the replication has limited replication
Eric> capabilities" is supposed to be "if the V-spoke PEs have limited
Eric> replication capabilities".   Since the V-hubs do more replication rather
Eric> than less, this "advantage" doesn't exist if the V-hubs have limited
Eric> replication capabilities.

Eric> Also, consider the following case.  There are 2 hubs and 10 spokes,
Eric> where each spoke is associated with both hubs.  Multicast is done using
Eric> a I-PMSI or (C-*,C-*) S-PMSI.  A source of (S,G) traffic is behind one
Eric> spoke, and each of the other spokes and each of the hubs has a receiver
Eric> for the (S,G) traffic.  Some of the spokes have joined (S,G) via one
Eric> hub, some via the other.  Now when the spoke attached to the source
Eric> gets an (S,G) packet from the customer site, it sends a replica to each
Eric> hub, and each hub sends a replica to each spoke.  That's a total of 22
Eric> replicas.  If the spoke had done the replication itself, there would be
Eric> only 11 replicas.  It's not obvious that this reduces any inefficiency.

3. Routing Information Exchange

   Routing information exchange among all the PEs of a given VPN is
   subject to the following rules.

Eric> It would be helpful to say which of the rules are normal RFC4364 rules,
Eric> and which are being introduced by this draft.

   A PE that has sites of a given VPN connected to it has to retain
   routing information received from the VPN sites connected to it.
   This is irrespective of whether this PE acts as a V-hub or a V-spoke
   of that VPN.

   A PE that has sites of a given VPN connected to it has to export (as
   VPN-IP routes) all the routes received from these sites.

Eric> This isn't precisely true.  It is often only the bestpaths that are
Eric> exported, not "all" the routes.  And sometimes routes are filtered so
Eric> that they are not exported.

   This is
   irrespective of whether this PE acts as a V-hub or a V-spoke of that
   VPN.

   In addition, a V-hub of a given VPN has to export a VPN-IP default

Eric> Should "has to" be "MUST"?

Eric> Is it really required to export a VPN-IP default route?  Wouldn't it be
Eric> sufficient to export a route that is a "summary route" of the VPN
Eric> customer's routes?  (Assuming that such a summary route were known?)
Eric> Or a small set of summary routes?  That should work, as long as the
Eric> label associated with the summary routes causes an IP lookup in the
Eric> VRF. 

   route for that VPN. This route SHOULD be exported to only the V-
   spokes of that VPN that are associated with that V-hub.

Eric> If this is "SHOULD" rather than "MUST", it would be worthwhile to
Eric> discuss  the downside of exporting the default route to the other
Eric> V-hubs. 

   To enable a V-spoke of a given VPN to share its outbound traffic load
   among the V-hubs associated with that V-spoke, each V-hub of that
   VPN, when originating a VPN-IP default route MUST use a distinct RD
   (per V-hub, per VPN).

   If a V-spoke imports several VPN-IP default routes, each originated
   by its own V-hub, and these routes have the same preference, then
   traffic from the V-spoke to other sites of that VPN would be load
   shared among the V-hubs.

   A V-hub of a given VPN has to import all the non-default VPN-IP
   routes originated by all other PEs that have sites of that VPN
   connected to them (irrespective of whether these other PEs act as V-
   hubs or V-spokes for that VPN).

Eric> Also mention explicitly that a V-hub has to import all the non-default
Eric> routes from V-spokes that are not associated with that V-hub.

   A V-hub of a given VPN MUST NOT import a VPN-IP default route unless
   this is the Internet VPN-IP default route (for the definition of
   "Internet VPN-IP default route" see section "Internet Connectivity").

Eric> Section "Internet Connectivity" does not answer the obvious question
Eric> here, which is how does a V-hub know whether the VPN-IP default route
Eric> it is importing is the Internet VPN-IP default route or not.

   A V-spoke of a given VPN has to import a VPN-IP default route for
   that VPN that has been originated by the V-hubs of that VPN that are
   associated with that V-spoke.

Eric> Suggest rewording to something like: "Within a given VPN, a V-spoke
Eric> MUST import all VPN-IP default routes that have been originated by the
Eric> V-hubs associated with that V-spoke".

   In addition, a V-spoke of a given VPN MAY import VPN-IP routes for
   that VPN that have been originated by some other V-spokes of that
   VPN.

Eric> May a V-spoke import VPN-IP routes from V-hubs that it is not
Eric> "associated" with?   May it import a VPN-IP route a V-spoke that is
Eric> associated with a different V-hub?  Is the intention that a V-spoke MAY
Eric> import any routes whatsoever, or are there some restrictions preventing
Eric> a V-spoke from importing certain routes?

   The above rules that govern the routing information exchange among
   all the PEs of a given VPN are realized using Route Target (RT)
   extended communities [rfc4360] and VRF export/import policies based
   on these RTs. One possible scheme that enables to realize such rules
   is described in Section "Deployment Considerations". This document
   does not preclude use of other schemes, as long as such schemes would
   support the above rules.

4. Forwarding Considerations

   This section describes changes/modifications to the forwarding
   procedures specified in [rfc4364].

   The MPLS label that the V-hub advertises with a VPN-IP default route
   MUST be the label that points to the VRF of the V-hub.  This way,
   whenever the V-hub receives packets carrying this label, the V-hub
   forwards the packets by performing IP lookup in the VRF identified by
   the label.

Eric> I think it is best to avoid phrases like "label that points to" or
Eric> "VRF identified by the label".  Such phrases have been known to
Eric> mislead people into thinking that labels have to be implemented as
Eric> pointers or identifiers of some sort.  I'd suggest saying something
Eric> like:  (a) the label is mapped to an NHLFE that identifies a
Eric> particular VRF, and (b) when the label is popped, the packet's
Eric> destination address is looked up in the VRF, (c) further disposition
Eric> of the packet is the result of this lookup (as specified later in this
Eric> document).

Eric> It should also be stated explicitly that "per-VRF labels" do NOT need
Eric> to be advertised for routes other than the default route.

   When a V-hub of a given VPN originates a VPN-IP default route for
   that VPN, the V-hub MUST NOT install in its VRF of that VPN a default
   route, unless this route has been originated either (a) as a result
   of the V-hub receiving an IP default route from one of the CEs of
   that VPN connected to it, or (b) as a result of the V-hub receiving
   (and importing) a VPN-IP default route from some other PE, or (c) the
   VRF being provisioned with a default route pointing to the routing
   table on the same PE that maintains the Internet routes.

Eric> I'm a little confused by this.  It seems to say that a V-hub that
Eric> originates an VPN-IP default route must not install a default route
Eric> (originated by someone else) unless that route is from a CE, or from
Eric> another PE, or else is a static route pointing to the global table.
Eric> What exactly is it that is NOT allowed?  The only thing that seems
Eric> prohibited is a static route to another VRF.  Is that the intention?

   In the absence of V-hubs and V-spokes, support for IBGP/EBGP load
   balancing in the presence of any-to-any connectivity uses the
   following rule.

Eric> Please include a reference to the RFC where this rule is given.  

   When a PE receives a packet from another PE, the PE
   that receives the  packet determines (using the MPLS label carried in
   the packet) the VRF that should be used for further forwarding the
   packet. If (using the information present in the VRF) the PE
   determines that the packet could be forwarded over one of the
   attachment circuits of that VRF (for the definition of "VRF
   attachment circuits" see [rfc4364]), then the PE MUST forward the
   packet over one of these circuits, and MUST NOT forward the packet to
   another PE.

Eric> This is the existing rule for load balancing?  To always forward the
Eric> traffic over a particular interface?  That doesn't sound like load
Eric> balancing to me.  Or do you really mean to say that IBGP/EBGP load
Eric> balancing is impossible according to the current L3VPN specs?  If the
Eric> latter, please supply the reference.

   In order to support IBGP/EBGP load balancing on a V-hub the above
   rule is modified as follows. When a V-hub receives a packet from
   another PE scenario, the V-hub identifies (using the MPLS label
   carried in the packet) the VRF that should be used for further
   forwarding the packet. If the MPLS label carried in the packet is the
   label that the V-hub advertised in the VPN-IP default route, then the
   V-hub is not restricted to use only one of the VRF attachment
   circuits for forwarding the packet - the V-hub may forward the packet
   to another PE.

Eric> If the V-hub may perform loadsharing by forwarding the packet to
Eric> another PE, how can we be sure that the other PE will not also perform
Eric> loadsharing and end up sending the packet back?  Perhaps it should be
Eric> required the label bound to a non-default route be a label that does
Eric> NOT cause an IP lookup in the VRF.  Or is it the intention that load
Eric> sharing only be done for data streams that area default routed?

   To illustrate this consider the following example:

                 <- RD:0/0           RD:0/0->

                                  <- RD:10.2.1        <-10.2.1/24
   CE1----PE-S-------------PE-H----------------PE-S1-------------CE2
                          /
                          |    |
                          |    |  10.2.1/24
                          |    |
                         CE4   CE3

   A multi-homed site (not shown in the above figure) is connect via CE2
   and CE3. Thus both CE2 and and CE3 advertise a route to 10.2.1/24.
   CE2 advertises this route to PE-S1, which in turn originates a VPN-IP
   route RD:10.2.1.  CE3 advertises this route to PE-H.

Eric> I guess "this route" in the last sentence is supposed to refer to the
Eric> IP route to 10.2.1/24, but the wording makes it seem like it refers to
Eric> the VPN-IP route to RD:10.2.1.

   PE-H is a V-hub, while PE-S and PE-S1 are V-spokes associated with
   that V-hub. Thus PE-H originates a VPN-IP default route (RD:0/0), and
   both PE-S and PE-S1 import that route.

   PE-H receives from PE-S1 a VPN-IP route to RD:10.2.1 and from CE3 a
   plain IP route to 10.2.1. Thus the VRF entry on PE-H has two possible
   next hops for 10.2.1: CE3 and PE-S1 (the latter is an indirect next
   hop).

Eric> Not sure what "indirect next hop" means in this context.     

   Now consider what happens when CE1 originates a packet destined to
   10.2.1.1. When PE-S receives this packet, PE-S (following the VPN-IP
   default route) forwards the packet to PE-H. In the absence of the
   modification specified above, when PE-H receives this packet, PE-H
   will be forced to forward the packet to CE3 (as PE-H can reach CE3
   via a VRF attachment circuit).  However, that would prevent IBGP/EBGP
   load balancing, as it would preclude the possibility of forwarding
   this packet via PE-S1 and CE2.

   With the modified rule specified above, PE-H determines that the MPLS
   label in the packet is the MPLS label that PE-H advertised to PE-S in
   the VPN-IP default route.

Eric> Perhaps replace this sentence with: "With the modified rule specified
Eric> above, the MPLS label in the packet is the label that PE-H advertised
Eric> to PE-S in the VPN-IP default route."  After all, when the packet
Eric> arrives, PE-H just looks up the label and follows the instructions in
Eric> the NHLFE; it doesn't determine how the label was advertised.

   Thus PE-H may forward the packet either via
   CE3 or via PE-S1 (with PE-S1 subsequently forwarding the packet to
   CE2), resulting in preserving IBGP/EBGP load balancing.

   Likewise, if CE4 originates a packet destined to 10.2.1.1, PE-H may
   forward the packet either via CE3 or via PE-S1 (with PE-S1
   subsequently forwarding the traffic to CE2), resulting in preserving
   IBGP/EBGP load balancing.

   Note however, that if there is some other CE, CE5, connected to PE-
   S1, and that CE5 sends a packet to 10.2.1.1, then (due to the IP
   longest match rule) PE-S1 will always forward this packet to CE2.
   Thus IBGP/EBGP load balancing will not be available for the
   destinations that have been learned locally by a V-spoke.

   Moreover, if CE3 advertises 10.2.1.0/25 and 10.2.1/24, while CE2
   advertises 10.2.1.128/25 and 10.2.1/24 (which is yet another form of
   load balancing for a multi-homed site), then when CE5 sends a packet
   to 10.2.1.1, then (due to the IP longest match rule) PE-S1 will
   always forward this packet to CE2, even though the VPN customer would
   expect this traffic to flow via CE3.

   This document proposes two options to address this, as well as to
   make the IBGP/EBGP load balancing available for the destinations that
   have been learned locally by a V-spoke. One option is to provision
   PEs that have multi-homed sites connected to them as V-hubs.

Eric> An SP can certainly have the policy that multi-homed customer sites do
Eric> not get connected to V-spokes.  However, it is not quite correct to say
Eric> that this policy "makes IBGP/EBGP load balancing available for the
Eric> destinations that have been learned locally by a V-spoke".  

   Another
   option is for the V-spoke, when it receives an IP route from a CE,
   not to install this route in its forwarding table, but just re-
   advertise this route as a VPN-IP route, together with an MPLS label.
   The packet's next hop of the Next Hop Label Forwarding Entry (NHLFE)
   [rfc3031] associated with that label MUST point to the CE that
   advertised the IP route.

Eric> Perhaps replace this last sentence with something like: "The NHLFE
Eric> associated with that label MUST specify that the CE that advertised the
Eric> IP route is the next hop".  

   As a result, when the PE receives data that
   carries that label, the PE performs no IP lookup on the data, and
   just forwards the data to the CE. Note that doing this would result
   in forcing the traffic between a pair of sites connected to the same
   V-spoke to go through the V-hub of that V-spoke.

Eric> This option does make "load balancing available for the destinations
Eric> that have been learned locally by a V-spoke", but the efficacy of the
Eric> load balancing seems questionable, as load due to traffic between the
Eric> two local sites is doubled.

5. Internet Connectivity

   In this document we consider two possible scenarios of providing
   Internet connectivity for a given VPN.

   The first scenario is when one or more sites of a given VPN are
   connected to a PE, and that PE also maintains Internet routes. In
   this case the Internet connectivity for that VPN is provided by
   provisioning in the VPN's VRF on that PE a default route pointing to
   the routing table on that PE that maintains the Internet routes.

Eric> Shouldn't this say "MAY be provided" rather than "is provided"?
   
   This PE MUST act as a V-hub for that VPN, and therefore MUST
   originate a VPN-IP default route.

Eric> I think what this really should say is that if a PE is acting as a
Eric> V-spoke for a particular VPN, then one MUST NOT provision the VPN's VRF
Eric> on that PE with a default route pointing to the routing table that
Eric> maintains the Internet routes".  As written, it suggests that doing
Eric> this provisioning will automatically change the PE from a V-spoke to a
Eric> V-hub.

   The second scenario is when a site of a given VPN has connection to
   the Internet, and a CE of that site advertises an IP default route to
   the PE connected to that CE. This scenario has two sub-cases: (a) PE
   is provisioned as a V-hub, and (b) PE is provisioned as a V-spoke.

   If a PE is provisioned as a V-hub, then the PE re-advertises this IP
   default route as a VPN-IP default route,

Eric> If the PE is provisioned as a V-hub, it presumably has already
Eric> advertised a VPN-IP default route.  Now it seems that it has to
Eric> advertise a new VPN-IP default route.  Please state exactly how the
Eric> new route and the old route differ.

   and install in its VRF an IP
   default route with the next hop pointing to the CE(s) that advertise
   the IP default route to the PE.

   If a PE is provisioned as a V-spoke, then the PE MUST NOT install in
   its VRF an IP default route.

Eric> If the V-spoke PE does not install an IP default route in its VRF, how
Eric> is traffic from the spoke site going to get to the hub?  Perhaps what
Eric> is meant is something like "receiving a default route from a CE does
Eric> not (or MUST NOT) cause a V-spoke to install a default route in its
Eric> VRF".

   The PE MUST originate a VPN-IP default
   route with a (non-null) MPLS label. The packet's next hop of the Next
   Hop Label Forwarding Entry (NHLFE) [rfc3031] associated with that
   label MUST point to the CE that advertises the IP default route. As a

Eric> I'd say that the next hop of the NHLFE "must be the CE" or "must
Eric> specify the CE", rather than "must point to the CE".  
   
   result, when the PE receives data that carries that label, the PE
   performs no IP lookup on the data, and just forwards the data to the
   CE. Note that in this case the VRF on the PE is going to have an IP
   default route, but this route would be created as a result of
   receiving a VPN-IP default route from one of the V-hubs of that PE
   (and not as a result of receiving the IP default route from the CE).
   Note also that if this PE has other sites of that VPN connected to
   it, then traffic from these sites to the Internet would go to that
   PE, then to the V-hub selected by the PE, then from that V-hub back
   to the PE, and then to the CE that advertises IP default route to the
   PE.

Eric> Would this ever be considered to be a desirable traffic pattern?  If a
Eric> customer gets his internet access via a gateway at a particular site,
Eric> perhaps it's not a very good idea to attach that site to a V-spoke.

   If a PE is provisioned as a V-spoke of a given VPN, and if a CE of
   that VPN advertises an IP default route to the PE (as the CE belongs
   to the site that provides the Internet connectivity for the VPN),
   then the PE can not advertise an IP default route back to that CE.
   Yet, the CE has to point to the PE as the next hop for all the
   traffic to other sites of that VPN. This document proposes the
   following options to accomplish this. The first option is to require
   the V-spoke to implement procedures of section "Further refinements".

Eric> Well, here's what it says in "further refinements":

           install in the V-spoke's data plane only the default route
           (following the Virtual Hub and Spoke model, as described above),

Eric> I think this means "install only the default route advertised by the
Eric> V-hub, not the default route advertised by the CE".  But the draft
Eric> really needs to make this clear.  (I think this also contradicts the
Eric> previous statement that the V-spoke does not install an IP default
Eric> route in its VRF; but perhaps it's just not clear what "install in the
Eric> V-spoke's data plane" means.)

           but keep all the VPN-IP routes in the V-spoke's control plane
           (and thus being able to advertise these routes from the V-spoke
           to the CEs).

Eric> If the V-spoke can hold all these control plane routes, why is it a
Eric> V-spoke and not a V-hub (or a "vanilla" PE)?  

   The second option is for the V-spoke to advertise to the CE 10/8,
   172.16/12, and 192.168/16. Note that this option is applicable only
   if the service provider knows that the VPN customer uses only the
   [rfc1918] addresses.

Eric> If it's known that the customer only uses RFC 1918 addresses, one can
Eric> have the V-hubs advertise VPN-IP routes 10/8, 172.16/12, and
Eric> 192.168/16, with per-VRF labels, instead of having them advertise
Eric> a VPN-IP default.  Then you wouldn't need any special configuration on
Eric> the V-spokes, and you wouldn't have to worry about having an IP default
Eric> route in the V-spoke VRF.

Eric> But I don't really see why it's important that the addresses be RFC1918
Eric> addresses.  If the SP knows the addresses used by the customer, and if
Eric> these addresses can be summarized by a small number of routes, it
Eric> doesn't make any difference whether the customer is using RFC1918
Eric> addresses or not.

   The third option is to re-provision this PE as a V-hub.

Eric> There are supposed to be three options to allow a PE provisioned as a
Eric> V-spoke to be connected to the customer site that provides Internet
Eric> connectivity for the customer.  This third option does not accomplish
Eric> its goal, since a PE provisioned as a V-hub is not a PE provisioned as
Eric> a V-spoke.  

   In all of the above we refer to the originated VPN-IP default route
   as the "Internet VPN-IP default route".

Eric> It is not clear what "all of the above" refers to.  I think it would
Eric> be clearer to say that an "Internet VPN-IP default route" is a VPN-IP
Eric> default route, advertised by a V-hub or V-spoke, but originated by a
Eric> CE attached to that V-hub or V-spoke.  Unless that's not correct, in
Eric> which case I think a clearer definition would be helpful.

   Note that if a V-hub originates the Internet VPN-IP default route for
   a given VPN, the V-hub does not originate any other VPN-IP default
   routes for that VPN.

Eric> Probably the V-hub has already originated such a route; this case
Eric> should be addressed explicitly.

   If a V-hub originates the Internet VPN-IP default route, then all the
   V-spokes associated with that V-hub MUST import that route. In
   addition (and in contrast with a non-Internet VPN-IP default route),
   other V-hubs MAY import that route.

Eric> But the draft hasn't said (at least not yet) how PEs other than the
Eric> originator can determine whether a VPN-IP default route is the
Eric> Internet VPN-IP default route or not!  Something to do with RTs, I
Eric> guess. 

   A V-hub MAY also import the Internet VPN-IP default routes originated
   by V-spoke(s).

   A V-spoke MUST NOT import the Internet VPN-IP default route
   originated by any other V-spoke. Such a route MAY be imported only by
   V-hubs.

   The above rules that govern exchange of the Internet VPN-IP default
   route among all the PEs of a given VPN are realized using Route
   Target (RT) extended communities [rfc4360] and VRF export/import
   policies based on these RTs. One possible scheme that enables to
   realize such rules is described in Section "Deployment
   Considerations".  This document does not preclude use of other
   schemes, as long as such schemes would support the above rules.

Eric> It seems to me that one of the necessary rules is that each VRF is
Eric> provisioned to know which RTs are carried by the Internet Default
Eric> VPN-IP route but not by the non-Internet Default VPN-IP route.
Eric> However, this rule is not mentioned anywhere in the draft.

   If the Internet VPN-IP default route originated by a V-hub has the
   same preference as the (non Internet) VPN-IP default route originated
   by some other V-hub, then a V-spoke that imports VPN-IP default
   routes originated by both of these V-hubs would load share the
   outgoing Internet traffic between these two V-hubs (and thus some of
   the outgoing Internet traffic from that V-spoke will first be routed
   to the V-hub that does not originate the Internet VPN-IP default
   route, and then from that V-hub to the V-hub that does originate the
   Internet VPN-IP default route).

   If taking an extra-hub hop for the Internet traffic is viewed as
   undesirable,

Eric> It's hard to imagine that this would ever be desirable, especially
Eric> since the "extra-hub hop" could cause the path to the Internet to be
Eric> arbitrarily lengthened.

   then it is RECOMMENDED that the Internet VPN-IP default
   route be of higher preference than a (non-Internet) VPN-IP default
   route originated by some other V-hub.

Eric> If the only way to tell whether a VPN-IP default route is Internet or
Eric> non-Internet is by looking at its route targets, following this
Eric> recommendation is going to require a very complicated set of policies
Eric> to be implemented, in order to handle all the various special cases
Eric> (e.g., the non-Internet default route is in the local AS, but the
Eric> Internet default route, which is to be preferable, is several ASes
Eric> away).   I have to wonder whether this is a realistic recommendation.

   However, in this case the
   traffic from the V-spokes to other sites of that VPN will not be load
   balanced. 

6. Deployment Considerations

   For a given VPN a V-hub and a set of V-spokes associated with that V-
   hub should be chosen in a way that minimizes the additional network
   distance/latency penalty, given that VPN geographic footprint.

   For a given VPN some/all of its V-spokes could be grouped into
   geographically-based clusters with any-any connectivity within each
   cluster. Note that the V-spokes within a given cluster need not be
   associated with the same V-hub(s). Likewise, not all V-spokes
   associated with a given V-hub need to be in the same cluster.  A use
   case for this would be VPN for large retail chain in which data
   traffic is hub/spoke between each store and centralized data-centers
   but there is need for direct VoIP traffic between stores within same
   geographical area.

Eric> I don't understand what it means to group a set of V-spokes into a
Eric> geographically-based cluster.  That is, what do you have to do exactly
Eric> in order to do this "grouping"? 

   The following describes one possible scheme that enables to realize
   the rules for exchanging routing information among PEs, as specified
   in Section "Routing Information Exchange".

   Consider a "vanilla" any-to-any VPN. All the PEs of such VPN are
   provisioned with the same export and import RT - we will refer to
   this RT as RT-VPN.

   To evolve this VPN into V-hubs and V-spokes we keep the same export
   RT-VPN on all the PEs of that VPN (both V-hubs and V-spokes). This
   RT-VPN is attached to all the VPN-IP routes that PEs originate as a
   result of receiving corresponding IP routes from their CEs.  We also
   keep the same import RT-VPN on all the V-hubs.

   In addition, each V-hub is provisioned with its own import RT, called
   RT-VH.  This RT-VH MUST be different from the export RT provisioned
   on that V-hub. For a given PE this RT-VH has to be distinct for every
   V-hub present on that PE. To avoid the situation where within a given
   VPN all the V-spokes would be associated with every V-hub (in other
   words, to partition V-spokes among V-hubs), different V-hubs within
   that VPN MAY use different RT-VHs. At one extreme every V-hub may use
   a distinct RT-VH. However, it is also possible for several V-hubs to
   use the same RT-VH, in which case all these V-hubs are used by the
   same set of V-spokes. Use of IP-address specific RTs may be an
   attractive option for RT-VHs.

Eric> If several V-hubs are to use the same RT-VH, then I don't see what is
Eric> attractive about the use of IP-address specific RTs.   

   When a V-hub originates a (non-Internet) VPN-IP default route, then
   the V-hub attaches RT-VH to that route. Thus this route is imported
   by all the V-spokes associated with the V-hub.

   If a V-hub originates the Internet VPN-IP default route, then the V-
   hub attaches both RT-VH and RT-VPN to that route. Thus this route is
   imported by all the V-spokes associated with the V-hub, as well as by
   other V-hubs.

   If a V-spoke originates the Internet VPN-IP default route, then the
   V-spoke attaches only RT-VPN to that route. Thus this route is not
   imported by any of the V-spokes, but is imported by V-hubs.

   A given V-spoke is provisioned to import only VPN-IP routes that
   carry RT-VHs of the V-hubs associated with that V-spoke (thus
   associating a new V-spoke with a given V-hub requires provisioning
   only on that V-spoke - no provisioning changes are required on the V-
   hub).

   A V-spoke may be provisioned to export VPN-IP routes not just to the
   V-hubs, but also to the V-spokes that import the same VPN-IP default
   route(s) as the V-spoke itself. The V-spoke accomplishes this by
   adding its import RT-VH(s) to the VPN-IP routes exported by the V-
   spoke.

   Use of RT Constrains [rfc4684] may further facilitate/optimize
   routing exchange in support of V-hubs and V-spokes.

Eric> The section on Internet Connectivity mentions several situations in
Eric> which a V-hub must know whether a default VPN-IP route it is receiving
Eric> is an Internet default VPN-IP route or not.  If the VPNs are set up
Eric> according to the deployment considerations of this section, how does a
Eric> V-hub make this determination?  Does each V-hub VRF have to be
Eric> explicitly configured to know that any VPN-IP default route carrying a
Eric> particular set of RTs (or all of a particular set, or some of a
Eric> particular set) is the Internet default route?  If so, this is not
Eric> simply a "deployment consideration", it is an essential part of the
Eric> V-hub/V-spoke architecture, and should be called out explicitly.

7. Multicast Considerations

   This section describes procedures for supporting MVPN in the presence
   of Virtual Hub-and-Spoke. The procedures rely on MVPN specifications
   as defined in [MVPN], [BGP-MVPN].

   The procedures assume that for the purpose of ensuring non-
   duplication both V-hubs and V-spokes can discard packets from a
   "wrong" PE, as specified in [MVPN]. This applies for all types of P-
   tunnels, including Ingress Replication.

7.1. Terminology

   When Ingress Replication is used to realize P-tunnels, a P-tunnel
   being "bound" to a particular I-PMSI/S-PMSI A-D route means the set
   of (unicast) tunnels used to carry traffic from the PE originating
   the route to the PEs that import the route.

   When Ingress Replication is used to realize P-tunnels, traffic
   received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A-
   D received by the PE means the traffic received on the (unicast)
   tunnel that the PE originating the route uses to send traffic to the
   PE that receives the route.

Eric> These sentences are difficult to parse.  Suggest something like:

       We will speak of a P-tunnel being "bound" to a particular
       I-PMSI/S-PMSI A-D route if the P-tunnel is specified in that route's
       PMSI Tunnel Attribute.

       When Ingress Replication is used, the P-tunnel bound to a particular
       I-PMSI/S-PMSI A-D route is actually a set of unicast tunnels.  The PE
       originating the route uses these unicast tunnels to carry traffic to
       the PEs that import the route.  When we say that traffic has been
       received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A-
       D route received by that PE, we refer to the unicast tunnel that the
       PE originating the route uses to send traffic to the PE that receives
       the route.

7.2. Eligible Upstream Multicast Hop (UMH) routes

   On a V-spoke the set of Eligible UMH routes consists of all the
   unicast VPN-IP routes received by the V-spoke, including the default
   VPN-IP routes received from its V-hub(s). Note that such routes may
   include VPN-IP routes received from other V-spokes.

Eric> This rules out the use of SAFI-129 routes for UMH determination.
Eric> Probably that is not the intention.   

7.3. Originating VPN-IP default route by a V-hub

   A V-hub, when originating a VPN-IP default route, follows the
   procedures of [BGP-MVPN].

Eric> You might refer more specifically to Sections 6 and 7 of [BGP-MVPN].  A
Eric> reference to Section 5.1 of [MVPN] might also be helpful.

   Specifically the V-hub must add the VRF
   Route Import extended community that embeds the V-hub's IP address.
   The route also must include the Source AS extended community.

7.4. Handling C-multicast routes

   Origination of C-multicast routes follow the procedures of [BGP-MVPN]
   (this is irrespective of whether these routes are originated by a V-
   hub or by a V-spoke). Note that for a given V-spoke its set of
   eligible UMH routes may contains not only the VPN-IP default routes
   advertised by its V-hubs, but also VPN-IP routes advertised by some
   other V-spokes.

Eric> Also, I think both Internet VPN-IP default routes and non-Internet
Eric> VPN-IP default routes are UMH-eligible; this would be worth pointing
Eric> out.

Eric> It's worth pointing out explicitly that in a V-spoke, no matter what
Eric> the multicast source is, the matching UMH-eligible route is probably
Eric> going to be the VPN-IP default route from one of the V-spoke's
Eric> associated V-hubs.  This means that the V-spoke is likely to see the
Eric> V-hub as the "upstream PE" for the multicast source.

   When a V-hub receives a C-multicast route, the V-hub determines
   whether the C-RP/C-S of the route is reachable via one of its VRF
   interfaces, and if yes, then the V-hub follows the [BGP-MVPN]
   procedures. Otherwise, the C-RP/C-S of the route is reachable via
   some other PE, in which case the V-hub uses the type (Source Tree
   Join vs Shared Tree Join), the Multicast Source, and Multicast Group
   from the received route to construct a new route.

Eric> suggest "construct a new route" --> "construct a new route of the same
Eric> type, with the same Multicast Source and Multicast Group"

Eric> I'd point out explicitly that this is the case where a V-spoke sees the
Eric> V-hub as the "upstream PE" for a given source, but the V-hub sees some
Eric> other PE (either V-hub or V-spoke) as the "upstream PE".  

   The hub constructs
   the rest of the new route following procedures specified in "11.1.3.
   Constructing the rest of the C-multicast route" of [BGP-MVPN]. The
   hub also creates the appropriate (C-*, C-G)/(C-S, C-G) state in its
   MVPN-TIB.

7.5. Originating I-PMSI/S-PMSI/SA A-D routes by V-spoke

   When a V-spoke originates an I-PMSI/S-PMSI/SA A-D route, the V-spoke
   follows the procedures of [BGP-MVPN], including the procedures for
   constructing RT(s) carried by the route. Note that as a result, such
   a route will be imported by the V-hubs. The P-tunnel bound to this
   route is used to carry to these V-hubs traffic originated by the
   sites connected to the V-spoke.

Eric> The above paragraph suggests, incorrectly, that an SA A-D route
Eric> identifies a P-tunnel

   This route may also be imported by some other V-spokes - the V-spokes
   that import the (unicast) VPN-IP routes originated by the V-spoke
   that originates the I-PMSI/S-PMSI/SA A-D route. In this case the
   route would carry not one, but two RTs: the first one results in
   importing the route by V-hubs, the second results in importing the
   route by the V-spokes (the second RT is the RT that V-spoke use for
   importing the VPN-IP default route). In this case the P-tunnel bound
   to this route is also used to carry traffic originated by the sites
   connected to the V-spoke that originates the route to these other V-
   spokes.

Eric> With regard to "not one, but two RTs", is there anything in this draft
Eric> that limits the number of RTs to two?

7.6. Originating I-PMSI/S-PMSI/SA A-D routes by V-hub

   When a V-hub originates an I-PMSI/S-PMSI/SA A-D route, the V-hub
   follows the procedures of [BGP-MVPN], except that in addition to the
   RT(s) constructed following these procedures, the route also carries
   the RT of the VPN-IP default route advertised by the V-hub.  Note
   that as a result, such a route will be imported by other V-hubs, and
   also by the V-spokes, but only by the V-spokes that import the VPN-IP
   default route originated by the V-hub.

Eric> It might be clearer to say "only by the V-spokes that are associated
Eric> with that V-hub".

   The P-tunnel bound to this
   route is used to carry to these other V-hubs and V-spokes the traffic
   originated by the sites connected to the V-hub that originates the
   route.

   In addition, a V-hub originates another I-PMSI A-D route - we'll
   refer to this route as a "Default I-PMSI A-D route".

Eric> I find this use of the word "Default" to be very confusing, as the
Eric> "Default I-PMSI A-D route" is not a "default route", nor is it a route
Eric> advertising a "default I-PMSI".  This I-PMSI A-D route only goes to
Eric> the V-hubs associated V-spokes, it might be less confusing to call it
Eric> the "Associated-V-spoke-only I-PMSI A-D route" or something like that.
Eric> (If the set of V-spokes associated with a given V-hub had a name,
Eric> e.g., "V-spoke cluster", this could be called an "Intra-cluster I-PMSI
Eric> A-D route", or a "cluster-specific I-PMSI A-D route", which would be
Eric> clearer.)

   The RT carried
   by this route is the RT that is carried in the VPN-IP default route
   advertised by the V-hub (RT-VH). Therefore, this route will be
   imported only by the V-spokes that import the VPN-IP default route
   advertised by this V-hub. The P-tunnel bound to this route is used to
   carry to these V-spokes traffic originated by either (a) other V-
   hubs, or (b) V-spokes, including the V-spokes that import the VPN-IP
   default route from the V-hub. More details on the use of this P-
   tunnel is described later.

   As a result, a V-hub originates not one, but two I-PMSI A-D routes.
   Each of these routes MUST have a distinct RD.

   When a V-hub receives traffic from one of the sites connected to the
   V-hub, and the V-hub determines (using some local policies) that this
   traffic should be transmitted using an I-PMSI, the V-hub forwards
   this traffic on the P-tunnel bound to the first (non Default) I-PMSI
   A-D route, but not on the P-tunnel bound to the second, the Default
   I-PMSI A-D route.

Eric> The use of the terms "first" and "second" above is confusing, as it is
Eric> not clear what ordering is being referred to.

7.7. Receiving I-PMSI/S-PMSI/SA A-D routes by V-spoke

   When a V-spoke receives an I-PMSI/S-PMSI/SA A-D route, the V-spoke
   follows the procedures of [BGP-MVPN]. As a result, a V-spoke that
   imports the VPN-IP default route originated by a given V-hub will
   also import I-PMSI/S-PMSI/SA A-D routes originated by that V-hub.
   Specifically, the V-spoke will import both the non Default I-PMSI A-D
   route and the Default I-PMSI A-D route originated by the V-hub.

   In addition, if a V-spoke imports the (unicast) VPN-IP routes
   originated by some other V-spokes, then the V-spoke will also import
   I-PMSI/S-PMSI/SA A-D routes originated by these other V-spokes.

7.8. Receiving I-PMSI/S-PMSI/SA A-D routes by V-hub

   The following describe procedures that a V-hub must follow when it
   receives an I-PMSI/S-PMSI/SA A-D route.

7.8.1. Case 1

   This is the case where the received I-PMSI/S-PMSI/SA A-D route was
   originated by a V-spoke that imports the VPN-IP default route originated
   by the V-hub, and the received route is imported not just by the V-hub
   (as well as by all other V-hubs), but also by other V-spokes, but only
   by the V-spokes that import the VPN-IP default route originated by the
   V-hub.

Eric> It's kind of hard to parse this sentence.  Does this mean that the
Eric> route was originated by one of the V-hub's associated V-spokes.  How
Eric> does the V-hub know that?  Does this also mean that the route is
Eric> imported by all the V-spokes associated with the V-hub?  How does the
Eric> V-hub know that?

   This is also the case where the received route was originated by
   a V-hub that uses the same RT as the received V-hub for advertising the

Eric> "received" --> "receiving"

   VPN-IP default route.

Eric> So the traffic transmitted in the P-tunnel bound to this route must be
Eric> traffic from a V-spoke site, right?  Other V-spokes associated with the
Eric> same hub may thus receive traffic directly from the originating
Eric> V-spoke, and all V-hubs (associated with the originating V-spoke or
Eric> not) receive traffic directly from the originating V-spoke.  Presumably
Eric> a V-hub, say V-hub2, that is not associated with the originating
Eric> V-spoke, will need to reforward traffic from the originating V-spoke to
Eric> the V-spokes with which it (V-hub2) is associated.
   
   In this case the received I-PMSI/S-PMSI/SA A-D route carries more
   than one RT. One of these RTs results in importing this route by the
   V-hubs. Another of these RTs is the RT that the V-hub uses when
   advertising its VPN-IP default route (RT-VH).

Eric> RT-VH is only mentioned in the "deployment considerations" section,
Eric> and setting up the RTs as described in that section is not mandatory.
Eric> The rules in this section have to be specified in a way that doesn't
Eric> presuppose on those deployment considerations.

   This RT results in
   importing the received I-PMSI/S-PMSI/SA A-D route o all the V-spokes
   that import the VPN-IP default route originated by the V-hub.

Eric> Since the V-hub has particular procedures to execute in this case, it
Eric> would be nice to say exactly how the V-hub knows that a particular
Eric> received A-D route is a "case 1" route.  Previous text suggests that
Eric> the only difference between the RT-VH and other RTs is the way the
Eric> RT-VH is configured to be an import or export RT of a particular set of
Eric> VRFs.  But this text suggests that the V-hub has to know which of its
Eric> import RTs is the RT-VH.

   In handling such I-PMSI/S-PMSI/SA A-D route the V-hub follows the
   [BGP-MVPN] procedures. Specifically, in contrast to Case 2 below, the
   V-hub MUST NOT re-advertise this route.

Eric> Is the paragraph below meant to be a summary of what it means to
Eric> "follow the [BGP-MVPN] procedures?   Or is it intended to specify some
Eric> new procedure?  I think the former, but that should be clarified.  The
Eric> [BGP-MVPN] procedure is that traffic traffic received from a P-tunnel
Eric> does not get re-forwarded onto another P-tunnel, right?

   The V-hub may forward traffic received on a P-tunnel bound to this I-
   PMSI/S-PMSI A-D route to only the sites connected to that V-hub.  The
   V-hub MUST NOT forward the traffic received on this P-tunnel to any
   other V-hubs or V-spokes, including the V-spokes that import the VPN-
   IP default route originated by the V-hub. Specifically, the V-hub
   MUST NOT forward the traffic received on the P-tunnel advertised in
   the received I-PMSI A-D route over the P-tunnel that the V-hub binds
   to its Default I-PMSI A-D route.

7.8.2. Case 2

   This is the case where the received I-PMSI/S-PMSI/SA A-D route was
   originated by either some other V-hub, or by a V-spoke. The I-PMSI/S-
   PMSI/SA A-D route is imported by the V-hub (as well as by other V-
   hubs), but not by any of the V-spokes that import the VPN-IP default
   route originated by the V-hub.

Eric> How does the V-hub know that the A-D route is not imported by any of
Eric> "the V-spokes that import the VPN-IP default route originated by the
Eric> V-hub"?  This rule should be specified in terms of the RTs carried by
Eric> the route, not in terms of what other PEs may or may not be doing.

Eric> BTW, I'd suggest using a term like "the V-spokes associated with the
Eric> V-hub" instead of "the V-spokes that import the VPN-IP default
Eric> route originated by the V-hub"

   In this case the received I-PMSI/S-PMSI/SA A-D route does not carry
   the RT that the receiving V-hub uses when advertising its VPN-IP
   default route (RT-VH), although the route may carry more than one RT.

Eric> Again, I think there should be a clear statement of how a V-hub knows,
Eric> by inspection of the RTs, when a received A-D route is a "case 2"
Eric> route.

   In this case the V-hubs follows the procedures of [BGP-MVPN] with the
   following additions.

   Once a V-hub accepts an I-PMSI A-D route, when the V-hub receives
   data on the P-tunnel bound to that I-PMSI A-D route, the V-hub
   follows procedures in [MVPN], [MVPN-BGP] to determine whether to
   accept the data. If the data is accepted, then the V-hub further
   forwards the data over the P-tunnel bound to the Default I-PMSI A-D
   route originated by the V-hub.

Eric> This seems to rule out the possibility of a V-hub using an "S-PMSI
Eric> only" model between a V-hub and its associated V-spokes, unless an
Eric> "S-PMSI only" model is used uniformly throughout the MVPN.  Is this the
Eric> intention?  If so, it should be stated explicitly.

   Note that in deciding whether to
   forward the data over the P-tunnel bound to the Default I-PMSI A-D
   route originated by the V-hub, the V-hub SHOULD take into account the
   (multicast) state present in its MVPN-TIB that has been created as a
   result receiving C-multicast routes from the V-spokes associated with
   the V-hub.

Eric> Certainly the V-hub should take its multicast forwarding state into
Eric> account when it does multicast forwarding ;-) Is something more
Eric> specific intended here?

   Whenever a V-hub accepts an S-PMSI/SA A-D route, the V-hub, in
   contrast to Case 1 above, MUST re-advertise this route to its V-
   spokes. To accomplish this, the V-hub changes the RT carried by the
   route to the RT that the V-hub uses when originating its VPN-IP
   default route (RT-VH),

Eric> What if the route is carrying multiple RTs?  Which one gets changed?
Eric> What happens to the others?   

   and sets Next Hop to self. In addition, for S-
   PMSI/SA A-D routes the V-hub changes the RD of the route to the RD
   that the hub uses when originating its VPN-IP default route, and for
   S-PMSI A-D routes the V-hub changes the Originating Router's IP
   address in the MCAST-VPN NLRI of the route to its own IP address.

   Moreover, before re-advertising S-PMSI A-D routes to V-spokes, the V-
   hub modifies the PMSI Tunnel attribute of these routes as appropriate
   (e.g., by replacing the P-tunnel rooted at the originator of these
   routes with a P-tunnel rooted at the V-hub).

   Whenever the V-hub receives data on the P-tunnel bound to the
   received S-PMSI A-D route, the V-hub follows procedures of [MVPN],
   [MVPN-BGP] to determine whether to accept the data. If the data is
   accepted, then the V-hub further forwards it over the P-tunnel bound
   to the S-PMSI A-D route that has been re-advertised by the V-hub.

Eric> At this point, there have been a lot of details, but still no
Eric> statement of the what the data path for multicast traffic is supposed
Eric> to be.  By reverse engineering the procedures, I conclude that the
Eric> intention is to do the following (please correct me if I am wrong).
Eric> Suppose V-spoke1 is associated with V-hub1, but not with V-hub2.
Eric> Suppose V-spoke2 is associated with V-hub2, but not with V-hub1.
Eric> Suppose there is a receiver, R, for (S,G) traffic, at a site attached
Eric> to V-spoke1.  And suppose that S is at a site attached either to
Eric> V-spoke2 or to V-hub2.  Now V-spoke1 will think the upstream PE for S
Eric> is V-hub1.  So V-spoke1 joins (S,G) via V-hub1, and V-hub1 then joins
Eric> (S,G) via V-spoke2 or V-hub2.  The data path of the (S,G) traffic that
Eric> goes to R will be S-->V-hub2/V-spoke2-->V-hub1-->V-spoke1-->R.

Eric> I.e., it's like real multicast, with V-hub1 as an intermediate hop.

Eric> I understand why V-spoke1 does not communicate in the control plane
Eric> with V-spoke2 or V-hub2.  But I don't really understand the point of
Eric> using V-hub1 as a intermediate hop in the data plane.  For unicast
Eric> traffic from a V-spoke, the associated V-hub needs to be an
Eric> intermediate hop, because it needs to do an IP lookup in its VRF.  For
Eric> multicast traffic, no IP lookup is done, and thus there is really no
Eric> need to pass the data traffic through the associated V-hub.

Eric> Let's consider the case where the source is at a site attached to
Eric> V-hub2.  V-hub1 will receive an S-PMSI A-D route matching (S,G) from
Eric> V-hub2.  V-hub1 then modifies this A-D route and forwards it to
Eric> V-spoke1.  V-hub1 could use this route to identify the P-tunnel
Eric> originating at V-hub2, thereby instructing V-spoke1 to join V-hub2's
Eric> tunnel directly.  Then V-hub1 would not be in the data path from S to
Eric> R, but it would participate in the control plane.  Wouldn't this meet
Eric> all the requirements of the V-hub/V-spoke architecture, while producing
Eric> a more optimal path for multicast data, and eliminating the need to
Eric> have the V-hubs splice together any P-tunnels?

Eric> Was any consideration given to such an alternative?

Eric> On another topic, this section should explicitly address any issues
Eric> that might arise if there are AS boundaries (or, if "seamless mpls' is
Eric> being used, area boundaries) between the hubs and spokes, or between
Eric> different hubs.  It shouldn't be left as an exercise to the reader to
Eric> figure out whether the procedures work unchanged in those casees. 

   If multiple S-PMSIs received by the V-hub have been aggregated into
   the same P-tunnel, then the V-hub, prior to forwarding to the V-
   spokes the data received on this P-tunnel may de-aggregate and then
   re-aggregate (in a different way) this data using the state present
   in its MVPN-TIB that has been created as a result of receiving C-
   multicast routes from the V-spokes. Even if S-PMSIs received by the
   V-hub each has its own P-tunnel, the V-hub, prior to forwarding to
   the V-spokes the data received on these P-tunnels may aggregate these
   S-PMSIs using the state present in its MVPN-TIB that has been created
   as a result of receiving C-multicast routes from the V-spokes.

7.9. Use of Ingress Replication with I-PMSI A-D routes

   The existing procedures for S-PMSI A-D routes are sufficient to
   discard packets coming from a "wrong" PE for any type of P-tunnels,
   including Ingress Replication.

   The following modifications to originating/receiving I-PMSI A-D
   routes enable to discard packets coming from a "wrong" PE when
   Ingress Replication is used for I-PMSI P-tunnels (for other types of
   P-tunnels the procedures specified in [MVPN], [BGP-MVPN] are
   sufficient).

   If Ingress Replication is used with I-PMSI A-D routes, then when a PE
   advertises such routes, the Tunnel Type in the PMSI Tunnel attribute
   is set to Ingress Replication; the Leaf Information Required flag is
   set to 1; the the attribute carries no MPLS labels.

Eric> I believe this is incompatible with RFC 6514, which prohibits the
Eric> setting of the Leaf Information Required flag in I-PMSI A-D routes.
Eric> This violates RFC 6514, which prohibits the setting fo the LIR flag in
Eric> I-PMSI A-D routes.

Eric> I do not see that it makes any sense to allow the LIR flag to be used
Eric> in I-PMSI A-D routes for the V-hub/V-spoke architecture, while
Eric> disallowing it otherwise.  I'm not even sure that's well-defined; how
Eric> exactly will a PE know whether an I-PMSI A-D route with LIR set is
Eric> legal in a particular scenario?

Eric> If it is considered desirable to allow the LIR flag to be used in
Eric> I-PMSI A-D routes, a separate draft should be written, updating RFC
Eric> 6514, that proposes to remove RFC 6514's restriction.  Presumably that
Eric> draft would address interoperability issues with implementations that
Eric> are compliant to RFC 6514 and disallow the LIR flag in I-PMSI A-D
Eric> routes.  However, I have to say that I don't think this change is
Eric> necessary, because the same effect can be obtained by using a wildcard
Eric> S-PMSI.

Eric> In any case, I don't think a change like this should be hidden in "the
Eric> fine print" of a draft that is on a different topic.

   A PE that receives such I-PMSI A-D route MUST respond with a Leaf A-D
   route.  The PMSI Tunnel attribute of that Leaf A-D route is
   constructed as follows. The Tunnel Type is set to Ingress
   Replication.  The Tunnel Identifier MUST carry a routable address of
   the PE that originates the Leaf A-D route. The PMSI Tunnel attribute
   MUST carry a downstream assigned MPLS label that is used to
   demultiplex the traffic received over a unicast tunnel by the PE.

...

9. Further refinements

   In some cases a VPN customer may not want to rely solely on an (IP)
   default route being advertised from a V-spoke to a CE, but may want
   CEs to receive all the VPN routes (e.g., for the purpose of faster
   detection of VPN connectivity failures, and activating some backup
   connectivity).

   In this case one possible approach would be to install in the V-
   spoke's data plane only the default route (following the Virtual Hub
   and Spoke model, as described above), but keep all the VPN-IP routes
   in the V-spoke's control plane (and thus being able to advertise
   these routes from the V-spoke to the CEs).  Granted, this would not
   change control plane resource consumption, but would (significantly)
   reduce resource consumption on the data plane.

Eric> If the whole point of the V-hub/V-spoke architecture is to allow
Eric> Spoke-PEs to maintain only a small set of routes, I don't quite see the
Eric> point of having the spoke-PE maintain all the Internet routes.