Subject: Re: Last Call comments on draft-ietf-l3vpn-virtual-hub

From: Yakov Rekhter <[email-masked]>
Date: Thu, 8 Nov 2012 07:17:19 -0800

Eric,

> I have a number of comments on the virtual-hub draft.  Some are minor and/or
> editorial, but a number are more substantial.  I think these comments need
> to be addressed before the draft is submitted for publication.
> 
> I've placed a lot of comments in-line, but let me summarize what I think are
> the major issues:

I am in the process of addressing your comments. In this e-mail I'd like
to focus on one particular one:

> Eric> Let's consider the case where the source is at a site attached to
> Eric> V-hub2.  V-hub1 will receive an S-PMSI A-D route matching (S,G) from
> Eric> V-hub2.  V-hub1 then modifies this A-D route and forwards it to
> Eric> V-spoke1.  V-hub1 could use this route to identify the P-tunnel
> Eric> originating at V-hub2, thereby instructing V-spoke1 to join V-hub2's
> Eric> tunnel directly.  Then V-hub1 would not be in the data path from S to
> Eric> R, but it would participate in the control plane.  Wouldn't this meet
> Eric> all the requirements of the V-hub/V-spoke architecture, while producing
> Eric> a more optimal path for multicast data, and eliminating the need to
> Eric> have the V-hubs splice together any P-tunnels?
> 
> Eric> Was any consideration given to such an alternative?

Please note that the procedures specified in the draft assume
the ability to perform sender-based RPF, as specified in 9.1.1
of rfc6513. Given that, if one would follow what you outlined above, 
could you point me to the specific text in 9.1.1 that would                
enable V-spoke1 to determine that from its own perspective the
UMH for (C-S, C-G) is V-hub1 ?

Furthermore, your outline above talks about S-PMSI A-D route. How
would it work for I-PMSI A-D routes ?

Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Mon, 19 Nov 2012 07:25:10 -0800

Eric,

> - It is not at all clear what has to be implemented in order to conform to
>   this draft.  There is a lot of complexity introduced to handle a few
>   specific scenarios, but it's not clear whether it is mandatory to support
>   those scenarios.  Is it mandatory to support the scenario where a customer
>   gets Internet access via a site that is homed to a V-spoke PE?  Is it
>   mandatory to support the scenario where a customer has a site that is
>   multihomed to a pair of V-spoke PEs, and the customer is expecting a
>   certain load-balancing behavior?
> 
>   Much of the discussion about these scenarios is of the sort that says "you
>   could do this, you could do that, or you could do some other thing", but
>   it's not clear what is required for conformance.  This really needs to be
>   clarified. 

Support the scenario where a customer gets Internet access via a site that 
is homed to a V-spoke PE is optional. The revised text (-03) makes this 
explicit.

> - Various procedures seem to assume that a PE, when processing a default
>   route, will know whether that route is "the Internet VPN-IP default route"
>   or "the non-Internet VPN-IP default route".  Yet there is no clear
>   statement of how this is known.  I think it is supposed to be inferred
>   somehow from the RTs carried by the route, but the draft never says that a
>   VRF has to be provisioned to know that any specific RT is used for this
>   particular purpose.

Done.

>   This is an example of a more general issue: the draft suggests that
>   certain routes need to be exported with a different RT (or a different set
>   of RTs) than certain other routes, but it doesn't say explicitly that an
>   importing VRF has to know that any particular import RT has any particular
>   semantics.  This is an area that I think needs to be clarified.

Done.

> - If V-hub1 receives a packet from the backbone, and the packet is destined
>   to a site which is attached both to V-hub1 and to another V-hub, V-hub2,
>   V-hub1 is allowed to do load-balancing by choosing whether (a) to send the
>   packet directly to the site, or (b) to send the packet to V-hub2.  But I
>   don't see anything in the draft that prevents V-hub2 from doing its own
>   load-balancing, and perhaps sending the packet back to V-hub1.

The possibility for V-hub(V-hub1) that has received an MPLS packet
from the backbone to send the packet to another PE (V-hub2) is
limited to packets received for the VPN-IP default route (identified
by a specific label). Such packet sent to V-hub2 has not matched a
VPN-IP default route on V-hub2 (since the hub does not import such
VPN-IP default route) but a more specific route. Hence V-hub2 will
not have the ability to send the packet to another PE. The revised
draft makes this explicit.

More on this below.

> - The draft says that a V-hub shouldn't originate a default route if it
>   receives a default route from one of its attached CEs.  But it doesn't say
>   what to do if the V-hub originates a default route and the CE later sends
>   it a default route.  Nor does it say what to do if the CE later withdraws
>   the default route.  One can surmise what needs to be done in these cases,
>   but the spec should say explicitly what to do.

Done.

> - In at least one place, the draft states "a PE that acts as a V-spoke of a
>   given VPN needs to maintain ... two or more VPN-IP default routes whose
>   Next Hops are the addresses of the V-hubs".  This seems to rule out the
>   case where the V-spoke and V-hub are separated by an "option B" ASBR.
>   (The option B ASBR would change the next hop.)  If this restriction is
>   intentional, it should be stated explicitly, and a reason for it should be
>   given.

Inter-AS is outside of the scope. The revised text makes this clear.

> - Hidden in the "fine print" is a significant change to RFC 6514.  This
>   draft proposes to allow the setting of the LIR flag in I-PMSI A-D routes,
>   while RFC 6514 prohibits it.  Presumably this change is only meant to
>   apply when the V-hub/V-spoke architecture is being used, but the draft
>   doesn't really say how the receiver of an I-PMSI A-D route is supposed to
>   know whether LIR is legal or not, and it doesn't address interoperability
>   issues with PEs that adhere to the RFC 6514 rules.  If this change is
>   considered desirable, it should really be separated out from this draft
>   and proposed in a separate internet-draft.

The draft assumes that when ingress replication is used all the PEs 
implement procedures specified in section 7.9 of the draft. The revised 
text makes this clear.

>   I think this should be removed from the V-hub draft.
> 
> - The V-hub/V-spoke architecture affects certain multicast scenarios, but not
>   others.  In particular, I think it only affects multicast scenarios where
>   (a) the source of a multicast flow is at a site attached to a V-spoke, and a
>   receiver of that flow is at a site attached to a different V-spoke, or (b)
>   the source of a multicast flow is at a site attached to a V-hub, and a
>   receiver of that flow is at a site attached to a V-spoke that is not
>   associated with that same V-hub.  But it is left as an exercise to the
>   reader to figure out what these scenarios are.  The draft should state
>   them explicitly.

Done.

> - In general,when a multicast flow needs to go from one V-spoke to another
>   V-spoke, the draft requires the BGP control messages to go from V-spoke to
>   V-hub to V-spoke.  I agree that, in order to meet the goals of the draft,
>   it is necessary to put the V-hub in the control path.  However, the draft
>   also requires the data flow itself to go from V-spoke to V-hub to V-spoke.
>   I do not see why the V-hub needs to insert itself into the multicast data
>   path.  For unicast, the V-hub needs to be in the data path because it
>   needs to do an IP lookup for each data packet.  For multicast, the V-hub
>   does not do an IP lookup for each data packet, so I don't see why it needs
>   to be in the data path.

The last statement in the above is incorrect - see 7.8.2 for counterexamples.
The revised draft made this explicit.

>   As long as the V-spoke PEs have global table unicast routes to each other,
>   it should be possible to keep the V-hubs out of the multicast data path
>   between two V-spokes.  (When a V-hub tells a V-spoke which P-tunnel to
>   join in order to receive (C-S,C-G) traffic, why can't the V-hub specify a
>   P-tunnel that is routed at the PE that is actually attached to the C-S?)
> 
>   The draft does not say that the V-spokes don't have global table routes to
>   each other, just that they don't have VPN-IP routes to the multicast
>   sources.  So I think the draft needs to provide some justification for
>   putting the V-hubs in the multicast data path.

see further down...

> - Something should be said about whether this draft is expected to work in a
>   "seamless MPLS environment" where there is an ABR between a V-hub and a
>   V-spoke.  Even a statement that seamless MPLS (and/or seamless MPLS
>   multicast) is out of scope (if that's the authors' intention) would be
>   helpful.  If "seamless MPLS multicast" is not out of scope, then something
>   needs to be said about the handling of the P2MP Segmented Next Hop
>   Extended Community on A-D routes received by the V-hub from V-spokes or
>   from other V-hubs.
> 
>   (It might also be helpful to say that "MVPN extranet" is out of scope.)

This draft is expected to work with the seamless MPLS unicast draft.
The two are orthogonal: seamless MPLS unicast draft deals with the
transport (MPLS LSP) layer while this draft deals with the VPN
service layer.

As you requested, I added a sentence to the draft stating that
seamless MPLS multicast is out of scope. 

> There are also a few more minor issues that I think need to be addressed:  
> 
> - I'd like to see some more explicit mention of the asymmetry in the
>   procedures.  A V-spoke only imports the default route from certain V-hubs,
>   but all V-hubs import all routes from all V-spokes.  I'm not saying that
>   this is a problem, just that it is a part of the architectural model and
>   should be mentioned explicitly.

Done.

> - Some may think the following is a very minor issue, but I think it is very
>   confusing to use the term "Default I-PMSI A-D route" to refer to a route
>   that is not a "default route" and that is also not a route advertising a
>   "default I-PMSI".  

Fixed.

> - The procedures of the draft do not appear to generalize easily to a
>   multi-level hierarchy.  If this is a non-goal, that should be stated
>   explicitly. 

There are plenty of non-goals. It is an explicit non-goal of the draft to
explicitly spell out all the non-goals.

> 
> 
> 
> More comments appear in-line.

responses in-line below....

> 
>   
> ...
> 
>                       draft-ietf-l3vpn-virtual-hub-01.txt
> 
> ...
> 
> 
> Abstract
> 
>    With BGP/MPLS VPNs any-to-any connectivity among sites of a given
>    Virtual Private Network would require each Provider Edge router that
>    has one or more of these sites connected to it to hold all the routes
>    of that Virtual Private Network. The approach described in this
>    document allows to reduce the number of Provider Edge routers that
>    have to maintain all these routes by requiring only a subset of these
>    routers to maintain all these routes.
> 
> Eric> As far as I can tell, the real purpose of this document is enable
> Eric> "BGP/MPLS IP VPNs" to provide any-to-any connectivity by using a sort
> Eric> of "hub and spoke" overlay.  Only the hubs hold all the routes.
> Eric> Spoke-to-spoke traffic is "default routed" from the source spoke to a
> Eric> hub, where it is reforwarded to the destination spoke.  I don't think
> Eric> the above paragraph captures this very well.

The "above paragraph" is in the Abstract section. As such, it need not go
into the details. If you look at section 2, it does go into a bit more
details and covers the points you mentioned above.

>    Furthermore, when Provider Edge routers use ingress replication to
>    carry multicast traffic of VPN customers, the approach described in
>    this document could allow to reduce bandwidth inefficiency associated
>    with ingress replication, and to redistribute the replication load
>    among Provider Edge routers.
> 
> Eric> I find this quite misleading; whether bandwidth inefficiency is
> Eric> reduced or increased depends upon the topology, and upon the location
> Eric> of the receivers relative to the location of the source and of the
> Eric> hubs.  

To address this, "could" is replaced with "may".

> Eric> Also, it's not clear what is meant by "redistribute the
> Eric> replication load", or why this is a good thing.

The drat does not make any value judgement on whether redistributing
the replication load "is a good thing".

> ...
> 
> 2. Overview
> 
>    With BGP/MPLS VPNs [rfc4364] providing any-to-any connectivity among
>    sites of a given Virtual Private Network (VPN) is usually
>    accomplished by requiring each Provider Edge router (PE) that has one
>    or more of these sites connected to it to hold all the routes of that
>    VPN.  The approach described in this document allows to reduce the
>    number of PEs that have to maintain all these routes by requiring
>    only a subset of such PEs to maintain all these routes.
> 
>    Consider a VPN that has its sites connected to a set of PEs. In the
>    context of this VPN we designate a subset of these PEs as "Virtual
>    Spoke" PEs (or just Virtual Spokes), while some other (non-
>    overlapping) subset of these PEs will be "Virtual Hub" PEs (or just
>    Virtual Hubs), with the rest of the PEs in the set will be "vanilla"
>    PEs (PEs that implement the [rfc4364] procedures, but do not
>    implement procedures specified in this document).
> 
>    For the sake of brevity we will use the term "V-hub" to denote a
>    Virtual Hub, and "V-spoke" to denote a Virtual Spoke.
> 
>    For a given VPN, its set of V-hubs could include not only the PEs
>    that have sites of that VPN connected to them, but also PEs that have
>    no sites of that VPN connected to them.
> 
> Eric> Two paragraphs before this, it says that a Virtual Hub or Spoke (of a
> Eric> given VPN) is a subset of the set of PEs that are attached to the sites
> Eric> of the VPN.  So how can a V-hub be a PE that has no sites of the VPN
> Eric> connected to it?

Yes, a V-hub can be a PE that has no sites connected to it. The revised
draft fixed this.

> Eric> Perhaps what is meant is that a particular V-hub VRF, say VRF-1, for a
> Eric> given VPN, may import routes from other V-hub and V-spoke VRFs of that
> Eric> VPN, even if VRF-1 is not itself connected to a customer site.

Correct. The revised draft fixes this.

> Eric> However, this does contradict the second paragraph of this section.

The revised draft fixes the second paragraph.

>    Note that while in the context of one VPN a given PE may act as a V-
>    hub, in the context of another VPN the same PE may act as a V-spoke,
>    and vice versa. Thus a given PE may act as a V-hub only for some, but
>    not all the VPNs present on that PE. Likewise, a given PE may act as
>    a V-spoke only for some, but not all the VPNs present on that PE.
> 
> Eric> It might have been better to define "V-hub" and "V-spoke" to be types
> Eric> of VRFs, rather than types of PEs.  (Then one wouldn't run into the
> Eric> awkwardness of having to say that a given PE is both a V-spoke PE and a
> Eric> V-hub PE, depending on which VPN one is speaking of.)

I am not sure that this would be better.

>    For a given VPN each V-spoke of that VPN is "associated" with two or
>    more V-hubs of that VPN (we use two V-hubs for redundancy to avoid a
>    single point of failure). Note that a given V-hub may have no V-
>    spokes associated with it.
> 
> Eric> Shouldn't it be "one or more"?  Presumably the procedures still work if
> Eric> a given V-spoke is associated with only one V-hub.  "Two or more" is
> Eric> just a deployment recommendation.

Correct. Fixed.

> Eric> Consider the set of V-spokes that are associated with V-hub-1.  Is it
> Eric> allowable for some V-spokes in the set to be associated with V-hub-2
> Eric> (and NOT associated with V-hub-3), while others are associated with
> Eric> V-hub-3 (and NOT associated with V-hub-2)?  I think the draft should
> Eric> answer this explicitly.

Done.

>    A PE that acts as a V-hub of a given VPN maintains all the routes of
>    that VPN.
> 
> Eric> By the way, this statement rules out the possibility of a multi-level
> Eric> hierarchy, where a VRF serves as a hub to the lower layer and as a
> Eric> spoke to the higher layer.  Is this intentionally ruled out?

Multi-level hierarchy is outside the scope.

>    A PE that acts as a V-spoke of a given VPN needs to
>    maintain only the routes originated by the sites of that VPN
>    connected to that PE,
> 
> Eric> suggest "only the routes originated by" --> "only the routes of that
> Eric> VPN that are originated by"

Done.

> 
>    plus two or more VPN-IP default routes whose
> 
> Eric> "one or more"?

Done.

> 
>    Next Hops are the addresses of the V-hubs associated with that V-
>    spoke.
> 
> Eric> An implication of this is that the V-hubs and associated V-spokes must
> Eric> be in the same AS.  Or at least that they are not separated by an ASBR
> Eric> with an "option B" interconnect?  Is that the intention?   (I don't
> Eric> think it is.)

Yes. The revised text makes this explicit.

>    This way, only a subset of PEs that have sites of a given VPN,
>    namely only the PEs acting as V-hubs of that VPN,  have to maintain
>    all the routes of that VPN. PEs acting as V-spokes of that VPN need
>    to maintain only a (small) subset of the routes of that VPN.
> 
>    When PEs use ingress replication to carry multicast traffic of VPN
>    customers, the approach described in this document could allow to
>    reduce bandwidth inefficiency associated with ingress replication,
>    and to redistribute the replication load among the PEs. This is
>    because a PE that acts as a V-spoke of a given VPN would need to
>    replicate multicast traffic only to other V-hubs (while other V-hubs
>    would replicate this traffic to the V-spokes associated with these V-
>    hubs), rather than to all the PEs of that VPN. Likewise, a PE that
>    acts as a V-hub of a given VPN would need to replicate multicast
>    traffic to other V-hubs, and the V-spokes, but only to the V-spokes
>    associated with that V-hub, rather than replicating the traffic to
>    all the PEs of that VPN. Limiting replication could be especially
>    beneficial if a PE that does the replication has limited replication
>    capabilities and/or has links with limited bandwidth.
> 
> Eric> I think "if a PE that does the replication has limited replication
> Eric> capabilities" is supposed to be "if the V-spoke PEs have limited
> Eric> replication capabilities".   Since the V-hubs do more replication rather
> Eric> than less, this "advantage" doesn't exist if the V-hubs have limited
> Eric> replication capabilities.

I changes the text, as you suggested.

> Eric> Also, consider the following case.  There are 2 hubs and 10 spokes,
> Eric> where each spoke is associated with both hubs.  Multicast is done using
> Eric> a I-PMSI or (C-*,C-*) S-PMSI.  A source of (S,G) traffic is behind one
> Eric> spoke, and each of the other spokes and each of the hubs has a receiver
> Eric> for the (S,G) traffic.  Some of the spokes have joined (S,G) via one
> Eric> hub, some via the other.  Now when the spoke attached to the source
> Eric> gets an (S,G) packet from the customer site, it sends a replica to each
> Eric> hub, and each hub sends a replica to each spoke.  That's a total of 22
> Eric> replicas.  If the spoke had done the replication itself, there would be
> Eric> only 11 replicas.  It's not obvious that this reduces any inefficiency.

And that is precisely why the revised text said that it *may*
reduce bandwidth inefficiency. 

> 3. Routing Information Exchange
> 
>    Routing information exchange among all the PEs of a given VPN is
>    subject to the following rules.
> 
> Eric> It would be helpful to say which of the rules are normal RFC4364 rules,
> Eric> and which are being introduced by this draft.

Done.

>    A PE that has sites of a given VPN connected to it has to retain
>    routing information received from the VPN sites connected to it.
>    This is irrespective of whether this PE acts as a V-hub or a V-spoke
>    of that VPN.
> 
>    A PE that has sites of a given VPN connected to it has to export (as
>    VPN-IP routes) all the routes received from these sites.
> 
> Eric> This isn't precisely true.  It is often only the bestpaths that are
> Eric> exported, not "all" the routes.  And sometimes routes are filtered so
> Eric> that they are not exported.

The rules of which routes to export follows the 4364 rules. The latest
draft makes this explicit.

>    This is
>    irrespective of whether this PE acts as a V-hub or a V-spoke of that
>    VPN.
> 
>    In addition, a V-hub of a given VPN has to export a VPN-IP default
> 
> Eric> Should "has to" be "MUST"?

Yes. Fixed.

> Eric> Is it really required to export a VPN-IP default route?  Wouldn't it be
> Eric> sufficient to export a route that is a "summary route" of the VPN
> Eric> customer's routes?  (Assuming that such a summary route were known?)
> Eric> Or a small set of summary routes?  That should work, as long as the
> Eric> label associated with the summary routes causes an IP lookup in the
> Eric> VRF. 

This spec requires to export a VPN-IP default route. 

>    route for that VPN. This route SHOULD be exported to only the V-
>    spokes of that VPN that are associated with that V-hub.
> 
> Eric> If this is "SHOULD" rather than "MUST", it would be worthwhile to
> Eric> discuss  the downside of exporting the default route to the other
> Eric> V-hubs. 

It should be "MUST". Fixed.

>    To enable a V-spoke of a given VPN to share its outbound traffic load
>    among the V-hubs associated with that V-spoke, each V-hub of that
>    VPN, when originating a VPN-IP default route MUST use a distinct RD
>    (per V-hub, per VPN).
> 
>    If a V-spoke imports several VPN-IP default routes, each originated
>    by its own V-hub, and these routes have the same preference, then
>    traffic from the V-spoke to other sites of that VPN would be load
>    shared among the V-hubs.
> 
>    A V-hub of a given VPN has to import all the non-default VPN-IP
>    routes originated by all other PEs that have sites of that VPN
>    connected to them (irrespective of whether these other PEs act as V-
>    hubs or V-spokes for that VPN).
> 
> Eric> Also mention explicitly that a V-hub has to import all the non-default
> Eric> routes from V-spokes that are not associated with that V-hub.

Done.

>    A V-hub of a given VPN MUST NOT import a VPN-IP default route unless
>    this is the Internet VPN-IP default route (for the definition of
>    "Internet VPN-IP default route" see section "Internet Connectivity").
> 
> Eric> Section "Internet Connectivity" does not answer the obvious question
> Eric> here, which is how does a V-hub know whether the VPN-IP default route
> Eric> it is importing is the Internet VPN-IP default route or not.

The new version answers this question in section "Internet Connectivity".

>    A V-spoke of a given VPN has to import a VPN-IP default route for
>    that VPN that has been originated by the V-hubs of that VPN that are
>    associated with that V-spoke.
> 
> Eric> Suggest rewording to something like: "Within a given VPN, a V-spoke
> Eric> MUST import all VPN-IP default routes that have been originated by the
> Eric> V-hubs associated with that V-spoke".

Done.

>    In addition, a V-spoke of a given VPN MAY import VPN-IP routes for
>    that VPN that have been originated by some other V-spokes of that
>    VPN.
> 
> Eric> May a V-spoke import VPN-IP routes from V-hubs that it is not
> Eric> "associated" with?   May it import a VPN-IP route a V-spoke that is
> Eric> associated with a different V-hub?  Is the intention that a V-spoke MAY
> Eric> import any routes whatsoever, or are there some restrictions preventing
> Eric> a V-spoke from importing certain routes?

The rules for export/import of VPN-IP routes are such that the only
routes a V-spoke may import are (a) the default VPN-IP route
originated by the V-hub(s) associated with the V-spoke, and (b)
VPN-IP routes originated by the V-spokes that are associated with
the same V-hub(s) as the V-spoke itself.  I clarified this in the
revised text.
  
>    The above rules that govern the routing information exchange among
>    all the PEs of a given VPN are realized using Route Target (RT)
>    extended communities [rfc4360] and VRF export/import policies based
>    on these RTs. One possible scheme that enables to realize such rules
>    is described in Section "Deployment Considerations". This document
>    does not preclude use of other schemes, as long as such schemes would
>    support the above rules.
> 
> 
> 4. Forwarding Considerations
> 
>    This section describes changes/modifications to the forwarding
>    procedures specified in [rfc4364].
> 
>    The MPLS label that the V-hub advertises with a VPN-IP default route
>    MUST be the label that points to the VRF of the V-hub.  This way,
>    whenever the V-hub receives packets carrying this label, the V-hub
>    forwards the packets by performing IP lookup in the VRF identified by
>    the label.
> 
> Eric> I think it is best to avoid phrases like "label that points to" or
> Eric> "VRF identified by the label".  Such phrases have been known to
> Eric> mislead people into thinking that labels have to be implemented as
> Eric> pointers or identifiers of some sort.  I'd suggest saying something
> Eric> like:  (a) the label is mapped to an NHLFE that identifies a
> Eric> particular VRF, and (b) when the label is popped, the packet's
> Eric> destination address is looked up in the VRF, (c) further disposition
> Eric> of the packet is the result of this lookup (as specified later in this
> Eric> document).

Done, as you suggested.

> Eric> It should also be stated explicitly that "per-VRF labels" do NOT need
> Eric> to be advertised for routes other than the default route.

Done, as you asked.

>    When a V-hub of a given VPN originates a VPN-IP default route for
>    that VPN, the V-hub MUST NOT install in its VRF of that VPN a default
>    route, unless this route has been originated either (a) as a result
>    of the V-hub receiving an IP default route from one of the CEs of
>    that VPN connected to it, or (b) as a result of the V-hub receiving
>    (and importing) a VPN-IP default route from some other PE, or (c) the
>    VRF being provisioned with a default route pointing to the routing
>    table on the same PE that maintains the Internet routes.
> 
> Eric> I'm a little confused by this.  It seems to say that a V-hub that
> Eric> originates an VPN-IP default route must not install a default route
> Eric> (originated by someone else) unless that route is from a CE, or from
> Eric> another PE, or else is a static route pointing to the global table.
> Eric> What exactly is it that is NOT allowed? The only thing that seems
> Eric> prohibited is a static route to another VRF.  Is that the intention?

Anything that is neither (a), nor (b), nor (c) is NOT allowed.

>    In the absence of V-hubs and V-spokes, support for IBGP/EBGP load
>    balancing in the presence of any-to-any connectivity uses the
>    following rule.
> 
> Eric> Please include a reference to the RFC where this rule is given.  
> 
>    When a PE receives a packet from another PE, the PE
>    that receives the  packet determines (using the MPLS label carried in
>    the packet) the VRF that should be used for further forwarding the
>    packet. If (using the information present in the VRF) the PE
>    determines that the packet could be forwarded over one of the
>    attachment circuits of that VRF (for the definition of "VRF
>    attachment circuits" see [rfc4364]), then the PE MUST forward the
>    packet over one of these circuits, and MUST NOT forward the packet to
>    another PE.
> 
> Eric> This is the existing rule for load balancing?  To always forward the
> Eric> traffic over a particular interface?  That doesn't sound like load
> Eric> balancing to me.  Or do you really mean to say that IBGP/EBGP load
> Eric> balancing is impossible according to the current L3VPN specs?  If the
> Eric> latter, please supply the reference.

In the absence of V-hubs and V-spokes support for IBGP/EBGP load
balancing is outside the scope of this draft. So, I removed the
above text (as it is clearly outside the scope of the draft).

>    In order to support IBGP/EBGP load balancing on a V-hub the above
>    rule is modified as follows. When a V-hub receives a packet from
>    another PE scenario, the V-hub identifies (using the MPLS label
>    carried in the packet) the VRF that should be used for further
>    forwarding the packet. If the MPLS label carried in the packet is the
>    label that the V-hub advertised in the VPN-IP default route, then the
>    V-hub is not restricted to use only one of the VRF attachment
>    circuits for forwarding the packet - the V-hub may forward the packet
>    to another PE.
> 
> Eric> If the V-hub may perform loadsharing by forwarding the packet to
> Eric> another PE, how can we be sure that the other PE will not also perform
> Eric> loadsharing and end up sending the packet back?  Perhaps it should be
> Eric> required the label bound to a non-default route be a label that does
> Eric> NOT cause an IP lookup in the VRF.  Or is it the intention that load
> Eric> sharing only be done for data streams that area default routed?

The latter. The new text spells this out.

>    To illustrate this consider the following example:
> 
>                  <- RD:0/0           RD:0/0->
> 
>                                   <- RD:10.2.1        <-10.2.1/24
>    CE1----PE-S-------------PE-H----------------PE-S1-------------CE2
>                           /
>                           |    |
>                           |    |  10.2.1/24
>                           |    |
>                          CE4   CE3
> 
>    A multi-homed site (not shown in the above figure) is connect via CE2
>    and CE3. Thus both CE2 and and CE3 advertise a route to 10.2.1/24.
>    CE2 advertises this route to PE-S1, which in turn originates a VPN-IP
>    route RD:10.2.1.  CE3 advertises this route to PE-H.
> 
> Eric> I guess "this route" in the last sentence is supposed to refer to the
> Eric> IP route to 10.2.1/24, but the wording makes it seem like it refers to
> Eric> the VPN-IP route to RD:10.2.1.

You are correct - "this route" refers to 10.2.1/24. Revised draft spells 
this out.

>    PE-H is a V-hub, while PE-S and PE-S1 are V-spokes associated with
>    that V-hub. Thus PE-H originates a VPN-IP default route (RD:0/0), and
>    both PE-S and PE-S1 import that route.
> 
>    PE-H receives from PE-S1 a VPN-IP route to RD:10.2.1 and from CE3 a
>    plain IP route to 10.2.1. Thus the VRF entry on PE-H has two possible
>    next hops for 10.2.1: CE3 and PE-S1 (the latter is an indirect next
>    hop).
> 
> Eric> Not sure what "indirect next hop" means in this context.     

Means PE-S1 is not directly connected to PE-H.

>    Now consider what happens when CE1 originates a packet destined to
>    10.2.1.1. When PE-S receives this packet, PE-S (following the VPN-IP
>    default route) forwards the packet to PE-H. In the absence of the
>    modification specified above, when PE-H receives this packet, PE-H
>    will be forced to forward the packet to CE3 (as PE-H can reach CE3
>    via a VRF attachment circuit).  However, that would prevent IBGP/EBGP
>    load balancing, as it would preclude the possibility of forwarding
>    this packet via PE-S1 and CE2.
> 
>    With the modified rule specified above, PE-H determines that the MPLS
>    label in the packet is the MPLS label that PE-H advertised to PE-S in
>    the VPN-IP default route.
> 
> Eric> Perhaps replace this sentence with: "With the modified rule specified
> Eric> above, the MPLS label in the packet is the label that PE-H advertised
> Eric> to PE-S in the VPN-IP default route."  After all, when the packet
> Eric> arrives, PE-H just looks up the label and follows the instructions in
> Eric> the NHLFE; it doesn't determine how the label was advertised.

Done, as you suggested.

>    Thus PE-H may forward the packet either via
>    CE3 or via PE-S1 (with PE-S1 subsequently forwarding the packet to
>    CE2), resulting in preserving IBGP/EBGP load balancing.
> 
>    Likewise, if CE4 originates a packet destined to 10.2.1.1, PE-H may
>    forward the packet either via CE3 or via PE-S1 (with PE-S1
>    subsequently forwarding the traffic to CE2), resulting in preserving
>    IBGP/EBGP load balancing.
> 
>    Note however, that if there is some other CE, CE5, connected to PE-
>    S1, and that CE5 sends a packet to 10.2.1.1, then (due to the IP
>    longest match rule) PE-S1 will always forward this packet to CE2.
>    Thus IBGP/EBGP load balancing will not be available for the
>    destinations that have been learned locally by a V-spoke.
> 
>    Moreover, if CE3 advertises 10.2.1.0/25 and 10.2.1/24, while CE2
>    advertises 10.2.1.128/25 and 10.2.1/24 (which is yet another form of
>    load balancing for a multi-homed site), then when CE5 sends a packet
>    to 10.2.1.1, then (due to the IP longest match rule) PE-S1 will
>    always forward this packet to CE2, even though the VPN customer would
>    expect this traffic to flow via CE3.
> 
>    This document proposes two options to address this, as well as to
>    make the IBGP/EBGP load balancing available for the destinations that
>    have been learned locally by a V-spoke. One option is to provision
>    PEs that have multi-homed sites connected to them as V-hubs.
> 
> Eric> An SP can certainly have the policy that multi-homed customer sites do
> Eric> not get connected to V-spokes.  However, it is not quite correct to say
> Eric> that this policy "makes IBGP/EBGP load balancing available for the
> Eric> destinations that have been learned locally by a V-spoke".  

Fixed.

>    Another
>    option is for the V-spoke, when it receives an IP route from a CE,
>    not to install this route in its forwarding table, but just re-
>    advertise this route as a VPN-IP route, together with an MPLS label.
>    The packet's next hop of the Next Hop Label Forwarding Entry (NHLFE)
>    [rfc3031] associated with that label MUST point to the CE that
>    advertised the IP route.
> 
> Eric> Perhaps replace this last sentence with something like: "The NHLFE
> Eric> associated with that label MUST specify that the CE that advertised the
> Eric> IP route is the next hop".  

Done, as you suggested.

>    As a result, when the PE receives data that
>    carries that label, the PE performs no IP lookup on the data, and
>    just forwards the data to the CE. Note that doing this would result
>    in forcing the traffic between a pair of sites connected to the same
>    V-spoke to go through the V-hub of that V-spoke.
> 
> Eric> This option does make "load balancing available for the destinations
> Eric> that have been learned locally by a V-spoke", but the efficacy of the
> Eric> load balancing seems questionable, as load due to traffic between the
> Eric> two local sites is doubled.

The load balancing refers to balancing the load on CE-PE links. Doubling
the traffic has to do with the provider infrastructure.

> 5. Internet Connectivity
> 
>    In this document we consider two possible scenarios of providing
>    Internet connectivity for a given VPN.
> 
>    The first scenario is when one or more sites of a given VPN are
>    connected to a PE, and that PE also maintains Internet routes. In
>    this case the Internet connectivity for that VPN is provided by
>    provisioning in the VPN's VRF on that PE a default route pointing to
>    the routing table on that PE that maintains the Internet routes.
> 
> Eric> Shouldn't this say "MAY be provided" rather than "is provided"?

Done, as you suggested.

>    This PE MUST act as a V-hub for that VPN, and therefore MUST
>    originate a VPN-IP default route.
> 
> Eric> I think what this really should say is that if a PE is acting as a
> Eric> V-spoke for a particular VPN, then one MUST NOT provision the VPN's VRF
> Eric> on that PE with a default route pointing to the routing table that
> Eric> maintains the Internet routes".  As written, it suggests that doing
> Eric> this provisioning will automatically change the PE from a V-spoke to a
> Eric> V-hub.

Done, as you suggested.

>    The second scenario is when a site of a given VPN has connection to
>    the Internet, and a CE of that site advertises an IP default route to
>    the PE connected to that CE. This scenario has two sub-cases: (a) PE
>    is provisioned as a V-hub, and (b) PE is provisioned as a V-spoke.
> 
>    If a PE is provisioned as a V-hub, then the PE re-advertises this IP
>    default route as a VPN-IP default route,
> 
> Eric> If the PE is provisioned as a V-hub, it presumably has already
> Eric> advertised a VPN-IP default route.  Now it seems that it has to
> Eric> advertise a new VPN-IP default route.  Please state exactly how the
> Eric> new route and the old route differ.

The two routes differ in their RTs. The revised text spells this out.

>    and install in its VRF an IP
>    default route with the next hop pointing to the CE(s) that advertise
>    the IP default route to the PE.
> 
>    If a PE is provisioned as a V-spoke, then the PE MUST NOT install in
>    its VRF an IP default route.
> 
> Eric> If the V-spoke PE does not install an IP default route in its VRF, how
> Eric> is traffic from the spoke site going to get to the hub?  Perhaps what
> Eric> is meant is something like "receiving a default route from a CE does
> Eric> not (or MUST NOT) cause a V-spoke to install a default route in its
> Eric> VRF".

You are correct. Done.

>    The PE MUST originate a VPN-IP default
>    route with a (non-null) MPLS label. The packet's next hop of the Next
>    Hop Label Forwarding Entry (NHLFE) [rfc3031] associated with that
>    label MUST point to the CE that advertises the IP default route. As a
> 
> Eric> I'd say that the next hop of the NHLFE "must be the CE" or "must
> Eric> specify the CE", rather than "must point to the CE".  

Done, as you suggested.

>    result, when the PE receives data that carries that label, the PE
>    performs no IP lookup on the data, and just forwards the data to the
>    CE. Note that in this case the VRF on the PE is going to have an IP
>    default route, but this route would be created as a result of
>    receiving a VPN-IP default route from one of the V-hubs of that PE
>    (and not as a result of receiving the IP default route from the CE).
>    Note also that if this PE has other sites of that VPN connected to
>    it, then traffic from these sites to the Internet would go to that
>    PE, then to the V-hub selected by the PE, then from that V-hub back
>    to the PE, and then to the CE that advertises IP default route to the
>    PE.
> 
> Eric> Would this ever be considered to be a desirable traffic pattern?  If a
> Eric> customer gets his internet access via a gateway at a particular site,
> Eric> perhaps it's not a very good idea to attach that site to a V-spoke.

Whether this is "a desirable traffic pattern", or whether "it's not
a very good idea" is outside the scope of the draft.

>    If a PE is provisioned as a V-spoke of a given VPN, and if a CE of
>    that VPN advertises an IP default route to the PE (as the CE belongs
>    to the site that provides the Internet connectivity for the VPN),
>    then the PE can not advertise an IP default route back to that CE.
>    Yet, the CE has to point to the PE as the next hop for all the
>    traffic to other sites of that VPN. This document proposes the
>    following options to accomplish this. The first option is to require
>    the V-spoke to implement procedures of section "Further refinements".
> 
> Eric> Well, here's what it says in "further refinements":
> 
>            install in the V-spoke's data plane only the default route
>            (following the Virtual Hub and Spoke model, as described above),
> 
> Eric> I think this means "install only the default route advertised by the
> Eric> V-hub, not the default route advertised by the CE".  But the draft
> Eric> really needs to make this clear.  (I think this also contradicts the

Done.

> Eric> previous statement that the V-spoke does not install an IP default
> Eric> route in its VRF; but perhaps it's just not clear what "install in the
> Eric> V-spoke's data plane" means.)
> 
>            but keep all the VPN-IP routes in the V-spoke's control plane
>            (and thus being able to advertise these routes from the V-spoke
>            to the CEs).
> 
> Eric> If the V-spoke can hold all these control plane routes, why is it a
> Eric> V-spoke and not a V-hub (or a "vanilla" PE)?  

Because the V-spoke need not hold these routes in its FIB.

>    The second option is for the V-spoke to advertise to the CE 10/8,
>    172.16/12, and 192.168/16. Note that this option is applicable only
>    if the service provider knows that the VPN customer uses only the
>    [rfc1918] addresses.
> 
> Eric> If it's known that the customer only uses RFC 1918 addresses, one can
> Eric> have the V-hubs advertise VPN-IP routes 10/8, 172.16/12, and
> Eric> 192.168/16, with per-VRF labels, instead of having them advertise
> Eric> a VPN-IP default.  Then you wouldn't need any special configuration on
> Eric> the V-spokes, and you wouldn't have to worry about having an IP default
> Eric> route in the V-spoke VRF.
> 
> Eric> But I don't really see why it's important that the addresses be RFC1918
> Eric> addresses.  If the SP knows the addresses used by the customer, and if
> Eric> these addresses can be summarized by a small number of routes, it
> Eric> doesn't make any difference whether the customer is using RFC1918
> Eric> addresses or not.

Assuming that the SP has to know the addresses used by a customer
may not be practical. This is irrespective of whether the customer
is using rfc1918 addresses or not. So, the second option is deleted
from the draft.

>    The third option is to re-provision this PE as a V-hub.
> 
> Eric> There are supposed to be three options to allow a PE provisioned as a
> Eric> V-spoke to be connected to the customer site that provides Internet
> Eric> connectivity for the customer.  This third option does not accomplish
> Eric> its goal, since a PE provisioned as a V-hub is not a PE provisioned as
> Eric> a V-spoke.  

That is correct. The revised text fixes this.

>    In all of the above we refer to the originated VPN-IP default route
>    as the "Internet VPN-IP default route".
> 
> Eric> It is not clear what "all of the above" refers to.  I think it would
> Eric> be clearer to say that an "Internet VPN-IP default route" is a VPN-IP
> Eric> default route, advertised by a V-hub or V-spoke, but originated by a
> Eric> CE attached to that V-hub or V-spoke.  

Done.

> Eric>                                        Unless that's not correct, in
> Eric> which case I think a clearer definition would be helpful.

>    Note that if a V-hub originates the Internet VPN-IP default route for
>    a given VPN, the V-hub does not originate any other VPN-IP default
>    routes for that VPN.
> 
> Eric> Probably the V-hub has already originated such a route; this case
> Eric> should be addressed explicitly.

Done.

>    If a V-hub originates the Internet VPN-IP default route, then all the
>    V-spokes associated with that V-hub MUST import that route. In
>    addition (and in contrast with a non-Internet VPN-IP default route),
>    other V-hubs MAY import that route.
> 
> Eric> But the draft hasn't said (at least not yet) how PEs other than the
> Eric> originator can determine whether a VPN-IP default route is the
> Eric> Internet VPN-IP default route or not!  Something to do with RTs, I
> Eric> guess. 

You guessed correctly. The revised text clarifies this.

> 
>    A V-hub MAY also import the Internet VPN-IP default routes originated
>    by V-spoke(s).
> 
>    A V-spoke MUST NOT import the Internet VPN-IP default route
>    originated by any other V-spoke. Such a route MAY be imported only by
>    V-hubs.
> 
>    The above rules that govern exchange of the Internet VPN-IP default
>    route among all the PEs of a given VPN are realized using Route
>    Target (RT) extended communities [rfc4360] and VRF export/import
>    policies based on these RTs. One possible scheme that enables to
>    realize such rules is described in Section "Deployment
>    Considerations".  This document does not preclude use of other
>    schemes, as long as such schemes would support the above rules.
> 
> Eric> It seems to me that one of the necessary rules is that each VRF is
> Eric> provisioned to know which RTs are carried by the Internet Default
> Eric> VPN-IP route but not by the non-Internet Default VPN-IP route.
> Eric> However, this rule is not mentioned anywhere in the draft.

The revised text fixes this.

>    If the Internet VPN-IP default route originated by a V-hub has the
>    same preference as the (non Internet) VPN-IP default route originated
>    by some other V-hub, then a V-spoke that imports VPN-IP default
>    routes originated by both of these V-hubs would load share the
>    outgoing Internet traffic between these two V-hubs (and thus some of
>    the outgoing Internet traffic from that V-spoke will first be routed
>    to the V-hub that does not originate the Internet VPN-IP default
>    route, and then from that V-hub to the V-hub that does originate the
>    Internet VPN-IP default route).
> 
>    If taking an extra-hub hop for the Internet traffic is viewed as
>    undesirable,
> 
> Eric> It's hard to imagine that this would ever be desirable, especially
> Eric> since the "extra-hub hop" could cause the path to the Internet to be
> Eric> arbitrarily lengthened.

Whether "it's hard to imagine that this would ever be desirable"
is outside the scope of the draft.

>    then it is RECOMMENDED that the Internet VPN-IP default
>    route be of higher preference than a (non-Internet) VPN-IP default
>    route originated by some other V-hub.
> 
> Eric> If the only way to tell whether a VPN-IP default route is Internet or
> Eric> non-Internet is by looking at its route targets, following this
> Eric> recommendation is going to require a very complicated set of policies
> Eric> to be implemented, in order to handle all the various special cases
> Eric> (e.g., the non-Internet default route is in the local AS, but the
> Eric> Internet default route, which is to be preferable, is several ASes
> Eric> away). I have to wonder whether this is a realistic recommendation.

Whether this is "a realistic recommendation" is outside the scope of the 
draft.

>    However, in this case the
>    traffic from the V-spokes to other sites of that VPN will not be load
>    balanced. 
> 
> 6. Deployment Considerations
> 
>    For a given VPN a V-hub and a set of V-spokes associated with that V-
>    hub should be chosen in a way that minimizes the additional network
>    distance/latency penalty, given that VPN geographic footprint.
> 
>    For a given VPN some/all of its V-spokes could be grouped into
>    geographically-based clusters with any-any connectivity within each
>    cluster. Note that the V-spokes within a given cluster need not be
>    associated with the same V-hub(s). Likewise, not all V-spokes
>    associated with a given V-hub need to be in the same cluster.  A use
>    case for this would be VPN for large retail chain in which data
>    traffic is hub/spoke between each store and centralized data-centers
>    but there is need for direct VoIP traffic between stores within same
>    geographical area.
> 
> Eric> I don't understand what it means to group a set of V-spokes into a
> Eric> geographically-based cluster.  That is, what do you have to do exactly
> Eric> in order to do this "grouping"? 

It means that the V-spokes are within the same geographical region.
The revised text makes this explicit.

>    The following describes one possible scheme that enables to realize
>    the rules for exchanging routing information among PEs, as specified
>    in Section "Routing Information Exchange".
> 
>    Consider a "vanilla" any-to-any VPN. All the PEs of such VPN are
>    provisioned with the same export and import RT - we will refer to
>    this RT as RT-VPN.
> 
>    To evolve this VPN into V-hubs and V-spokes we keep the same export
>    RT-VPN on all the PEs of that VPN (both V-hubs and V-spokes). This
>    RT-VPN is attached to all the VPN-IP routes that PEs originate as a
>    result of receiving corresponding IP routes from their CEs.  We also
>    keep the same import RT-VPN on all the V-hubs.
> 
>    In addition, each V-hub is provisioned with its own import RT, called

There is a bug here - it should be "its own export RT", not "its own import
RT".

>    RT-VH.  This RT-VH MUST be different from the export RT provisioned
>    on that V-hub. For a given PE this RT-VH has to be distinct for every
>    V-hub present on that PE. To avoid the situation where within a given
>    VPN all the V-spokes would be associated with every V-hub (in other
>    words, to partition V-spokes among V-hubs), different V-hubs within
>    that VPN MAY use different RT-VHs. At one extreme every V-hub may use
>    a distinct RT-VH. However, it is also possible for several V-hubs to
>    use the same RT-VH, in which case all these V-hubs are used by the
>    same set of V-spokes. Use of IP-address specific RTs may be an
>    attractive option for RT-VHs.
> 
> Eric> If several V-hubs are to use the same RT-VH, then I don't see what is
> Eric> attractive about the use of IP-address specific RTs.   

The revised text clarifies that the use of IP-address specific RTs may be
an attractive option when every V-hub uses a distinct RT-VH.

>    When a V-hub originates a (non-Internet) VPN-IP default route, then
>    the V-hub attaches RT-VH to that route. Thus this route is imported
>    by all the V-spokes associated with the V-hub.
> 
> 
>    If a V-hub originates the Internet VPN-IP default route, then the V-
>    hub attaches both RT-VH and RT-VPN to that route. Thus this route is
>    imported by all the V-spokes associated with the V-hub, as well as by
>    other V-hubs.
> 
>    If a V-spoke originates the Internet VPN-IP default route, then the
>    V-spoke attaches only RT-VPN to that route. Thus this route is not
>    imported by any of the V-spokes, but is imported by V-hubs.
> 
>    A given V-spoke is provisioned to import only VPN-IP routes that
>    carry RT-VHs of the V-hubs associated with that V-spoke (thus
>    associating a new V-spoke with a given V-hub requires provisioning
>    only on that V-spoke - no provisioning changes are required on the V-
>    hub).
> 
>    A V-spoke may be provisioned to export VPN-IP routes not just to the
>    V-hubs, but also to the V-spokes that import the same VPN-IP default
>    route(s) as the V-spoke itself. The V-spoke accomplishes this by
>    adding its import RT-VH(s) to the VPN-IP routes exported by the V-
>    spoke.
> 
>    Use of RT Constrains [rfc4684] may further facilitate/optimize
>    routing exchange in support of V-hubs and V-spokes.
> 
> Eric> The section on Internet Connectivity mentions several situations in
> Eric> which a V-hub must know whether a default VPN-IP route it is receiving
> Eric> is an Internet default VPN-IP route or not.  
> Eric> If the VPNs are set up
> Eric> according to the deployment considerations of this section, how does a
> Eric> V-hub make this determination?  Does each V-hub VRF have to be
> Eric> explicitly configured to know that any VPN-IP default route carrying a
> Eric> particular set of RTs (or all of a particular set, or some of a
> Eric> particular set) is the Internet default route?  

Yes.

> Eric> If so, this is not
> Eric> simply a "deployment consideration", it is an essential part of the
> Eric> V-hub/V-spoke architecture, and should be called out explicitly.

Done, as you requested.

> 
> 
> 
> 
> 7. Multicast Considerations
> 
>    This section describes procedures for supporting MVPN in the presence
>    of Virtual Hub-and-Spoke. The procedures rely on MVPN specifications
>    as defined in [MVPN], [BGP-MVPN].
> 
>    The procedures assume that for the purpose of ensuring non-
>    duplication both V-hubs and V-spokes can discard packets from a
>    "wrong" PE, as specified in [MVPN]. This applies for all types of P-
>    tunnels, including Ingress Replication.
> 
> 
> 7.1. Terminology
> 
>    When Ingress Replication is used to realize P-tunnels, a P-tunnel
>    being "bound" to a particular I-PMSI/S-PMSI A-D route means the set
>    of (unicast) tunnels used to carry traffic from the PE originating
>    the route to the PEs that import the route.
> 
>    When Ingress Replication is used to realize P-tunnels, traffic
>    received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A-
>    D received by the PE means the traffic received on the (unicast)
>    tunnel that the PE originating the route uses to send traffic to the
>    PE that receives the route.
> 
> Eric> These sentences are difficult to parse.  Suggest something like:
>        We will speak of a P-tunnel being "bound" to a particular
>        I-PMSI/S-PMSI A-D route if the P-tunnel is specified in that route's
>        PMSI Tunnel Attribute.
> 
>        When Ingress Replication is used, the P-tunnel bound to a particular
>        I-PMSI/S-PMSI A-D route is actually a set of unicast tunnels.  The PE
>        originating the route uses these unicast tunnels to carry traffic to
>        the PEs that import the route.  When we say that traffic has been
>        received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A-
>        D route received by that PE, we refer to the unicast tunnel that the
>        PE originating the route uses to send traffic to the PE that receives
>        the route.

Done, as you suggested.

> 7.2. Eligible Upstream Multicast Hop (UMH) routes
> 
>    On a V-spoke the set of Eligible UMH routes consists of all the
>    unicast VPN-IP routes received by the V-spoke, including the default
>    VPN-IP routes received from its V-hub(s). Note that such routes may
>    include VPN-IP routes received from other V-spokes.
> 
> Eric> This rules out the use of SAFI-129 routes for UMH determination.
> Eric> Probably that is not the intention.   

You are correct - this is not the intention. The revised text spells this out.
 
> 7.3. Originating VPN-IP default route by a V-hub
> 
>    A V-hub, when originating a VPN-IP default route, follows the
>    procedures of [BGP-MVPN].
> 
> Eric> You might refer more specifically to Sections 6 and 7 of [BGP-MVPN].  A
> Eric> reference to Section 5.1 of [MVPN] might also be helpful.

Done.

> 
>    Specifically the V-hub must add the VRF
>    Route Import extended community that embeds the V-hub's IP address.
>    The route also must include the Source AS extended community.
> 
> 
> 7.4. Handling C-multicast routes
> 
>    Origination of C-multicast routes follow the procedures of [BGP-MVPN]
>    (this is irrespective of whether these routes are originated by a V-
>    hub or by a V-spoke). Note that for a given V-spoke its set of
>    eligible UMH routes may contains not only the VPN-IP default routes
>    advertised by its V-hubs, but also VPN-IP routes advertised by some
>    other V-spokes.
> 
> Eric> Also, I think both Internet VPN-IP default routes and non-Internet
> Eric> VPN-IP default routes are UMH-eligible; this would be worth pointing
> Eric> out.

I fixed this in 7.2 of the latest draft.

> Eric> It's worth pointing out explicitly that in a V-spoke, no matter what
> Eric> the multicast source is, the matching UMH-eligible route is probably
> Eric> going to be the VPN-IP default route from one of the V-spoke's
> Eric> associated V-hubs.  This means that the V-spoke is likely to see the
> Eric> V-hub as the "upstream PE" for the multicast source.

That is not always the case, as a V-spoke may receive UMH-eligible routes 
from other V-spokes, in which case the V-hub will *not* be the upstream
PE for the multicast sources connected to these V-spokes.

>    When a V-hub receives a C-multicast route, the V-hub determines
>    whether the C-RP/C-S of the route is reachable via one of its VRF
>    interfaces, and if yes, then the V-hub follows the [BGP-MVPN]
>    procedures. Otherwise, the C-RP/C-S of the route is reachable via
>    some other PE, in which case the V-hub uses the type (Source Tree
>    Join vs Shared Tree Join), the Multicast Source, and Multicast Group
>    from the received route to construct a new route.
> 
> Eric> suggest "construct a new route" --> "construct a new route of the same
> Eric> type, with the same Multicast Source and Multicast Group"

Done, as you suggested.

> Eric> I'd point out explicitly that this is the case where a V-spoke sees the
> Eric> V-hub as the "upstream PE" for a given source, but the V-hub sees some
> Eric> other PE (either V-hub or V-spoke) as the "upstream PE".  

Done.

> 
>    The hub constructs
>    the rest of the new route following procedures specified in "11.1.3.
>    Constructing the rest of the C-multicast route" of [BGP-MVPN]. The
>    hub also creates the appropriate (C-*, C-G)/(C-S, C-G) state in its
>    MVPN-TIB.
> 
> 
> 7.5. Originating I-PMSI/S-PMSI/SA A-D routes by V-spoke
> 
>    When a V-spoke originates an I-PMSI/S-PMSI/SA A-D route, the V-spoke
>    follows the procedures of [BGP-MVPN], including the procedures for
>    constructing RT(s) carried by the route. Note that as a result, such
>    a route will be imported by the V-hubs. The P-tunnel bound to this
>    route is used to carry to these V-hubs traffic originated by the
>    sites connected to the V-spoke.
> 
> Eric> The above paragraph suggests, incorrectly, that an SA A-D route
> Eric> identifies a P-tunnel

The revised text clarifies this.

> 
>    This route may also be imported by some other V-spokes - the V-spokes
>    that import the (unicast) VPN-IP routes originated by the V-spoke
>    that originates the I-PMSI/S-PMSI/SA A-D route. In this case the
>    route would carry not one, but two RTs: the first one results in
>    importing the route by V-hubs, the second results in importing the
>    route by the V-spokes (the second RT is the RT that V-spoke use for
>    importing the VPN-IP default route). In this case the P-tunnel bound
>    to this route is also used to carry traffic originated by the sites
>    connected to the V-spoke that originates the route to these other V-
>    spokes.
> 
> Eric> With regard to "not one, but two RTs", is there anything in this draft
> Eric> that limits the number of RTs to two?

No. Fixed.

> 7.6. Originating I-PMSI/S-PMSI/SA A-D routes by V-hub
> 
>    When a V-hub originates an I-PMSI/S-PMSI/SA A-D route, the V-hub
>    follows the procedures of [BGP-MVPN], except that in addition to the
>    RT(s) constructed following these procedures, the route also carries
>    the RT of the VPN-IP default route advertised by the V-hub.  Note
>    that as a result, such a route will be imported by other V-hubs, and
>    also by the V-spokes, but only by the V-spokes that import the VPN-IP
>    default route originated by the V-hub.
> 
> Eric> It might be clearer to say "only by the V-spokes that are associated
> Eric> with that V-hub".

Done.

> 
>    The P-tunnel bound to this
>    route is used to carry to these other V-hubs and V-spokes the traffic
>    originated by the sites connected to the V-hub that originates the
>    route.
> 
>    In addition, a V-hub originates another I-PMSI A-D route - we'll
>    refer to this route as a "Default I-PMSI A-D route".
> 
> Eric> I find this use of the word "Default" to be very confusing, as the
> Eric> "Default I-PMSI A-D route" is not a "default route", nor is it a route
> Eric> advertising a "default I-PMSI".  This I-PMSI A-D route only goes to
> Eric> the V-hubs associated V-spokes, it might be less confusing to call it
> Eric> the "Associated-V-spoke-only I-PMSI A-D route" or something like that.
> Eric> (If the set of V-spokes associated with a given V-hub had a name,
> Eric> e.g., "V-spoke cluster", this could be called an "Intra-cluster I-PMSI
> Eric> A-D route", or a "cluster-specific I-PMSI A-D route", which would be
> Eric> clearer.)

The revised draft uses the term "Associated V-spoke-only I-PMSI A-D route".

>    The RT carried
>    by this route is the RT that is carried in the VPN-IP default route
>    advertised by the V-hub (RT-VH). Therefore, this route will be
>    imported only by the V-spokes that import the VPN-IP default route
>    advertised by this V-hub. The P-tunnel bound to this route is used to
>    carry to these V-spokes traffic originated by either (a) other V-
>    hubs, or (b) V-spokes, including the V-spokes that import the VPN-IP
>    default route from the V-hub. More details on the use of this P-
>    tunnel is described later.
> 
>    As a result, a V-hub originates not one, but two I-PMSI A-D routes.
>    Each of these routes MUST have a distinct RD.
> 
>    When a V-hub receives traffic from one of the sites connected to the
>    V-hub, and the V-hub determines (using some local policies) that this
>    traffic should be transmitted using an I-PMSI, the V-hub forwards
>    this traffic on the P-tunnel bound to the first (non Default) I-PMSI
>    A-D route, but not on the P-tunnel bound to the second, the Default
>    I-PMSI A-D route.
> 
> Eric> The use of the terms "first" and "second" above is confusing, as it is
> Eric> not clear what ordering is being referred to.

Done.

> 
> 7.7. Receiving I-PMSI/S-PMSI/SA A-D routes by V-spoke
> 
>    When a V-spoke receives an I-PMSI/S-PMSI/SA A-D route, the V-spoke
>    follows the procedures of [BGP-MVPN]. As a result, a V-spoke that
>    imports the VPN-IP default route originated by a given V-hub will
>    also import I-PMSI/S-PMSI/SA A-D routes originated by that V-hub.
>    Specifically, the V-spoke will import both the non Default I-PMSI A-D
>    route and the Default I-PMSI A-D route originated by the V-hub.
> 
>    In addition, if a V-spoke imports the (unicast) VPN-IP routes
>    originated by some other V-spokes, then the V-spoke will also import
>    I-PMSI/S-PMSI/SA A-D routes originated by these other V-spokes.
> 
> 
> 7.8. Receiving I-PMSI/S-PMSI/SA A-D routes by V-hub
> 
>    The following describe procedures that a V-hub must follow when it
>    receives an I-PMSI/S-PMSI/SA A-D route.
> 
> 
> 7.8.1. Case 1
> 
>    This is the case where the received I-PMSI/S-PMSI/SA A-D route was
>    originated by a V-spoke that imports the VPN-IP default route originated
>    by the V-hub, and the received route is imported not just by the V-hub
>    (as well as by all other V-hubs), but also by other V-spokes, but only
>    by the V-spokes that import the VPN-IP default route originated by the
>    V-hub.
> 
> Eric> It's kind of hard to parse this sentence.  Does this mean that the
> Eric> route was originated by one of the V-hub's associated V-spokes.  

Correct. 

> Eric> How does the V-hub know that?  

By examining the RT(s) carried in the route.

> Eric> Does this also mean that the route is
> Eric> imported by all the V-spokes associated with the V-hub?  How does the
> Eric> V-hub know that?

Yes. The V-hub knows that by examining the RT(s) carried in the route.

>    This is also the case where the received route was originated by
>    a V-hub that uses the same RT as the received V-hub for advertising the
> 
> Eric> "received" --> "receiving"

Done.

> 
>    VPN-IP default route.
> 
> 
> Eric> So the traffic transmitted in the P-tunnel bound to this route must be
> Eric> traffic from a V-spoke site, right?  

It could be either from a V-spoke site, or from a V-hub site.

> Eric> Other V-spokes associated with the
> Eric> same hub may thus receive traffic directly from the originating
> Eric> V-spoke, and all V-hubs (associated with the originating V-spoke or
> Eric> not) receive traffic directly from the originating V-spoke.  Presumably
> Eric> a V-hub, say V-hub2, that is not associated with the originating
> Eric> V-spoke, will need to reforward traffic from the originating V-spoke to
> Eric> the V-spokes with which it (V-hub2) is associated.

The case where V-hub2 has to reforward traffic is covered in Case 2.

>    In this case the received I-PMSI/S-PMSI/SA A-D route carries more
>    than one RT. One of these RTs results in importing this route by the
>    V-hubs. Another of these RTs is the RT that the V-hub uses when
>    advertising its VPN-IP default route (RT-VH).
>
> Eric> RT-VH is only mentioned in the "deployment considerations" section,
> Eric> and setting up the RTs as described in that section is not mandatory.
> Eric> The rules in this section have to be specified in a way that doesn't
> Eric> presuppose on those deployment considerations.

The revised text fixes this.

>    This RT results in
>    importing the received I-PMSI/S-PMSI/SA A-D route o all the V-spokes
>    that import the VPN-IP default route originated by the V-hub.
> 
> Eric> Since the V-hub has particular procedures to execute in this case, it
> Eric> would be nice to say exactly how the V-hub knows that a particular
> Eric> received A-D route is a "case 1" route.  Previous text suggests that
> Eric> the only difference between the RT-VH and other RTs is the way the
> Eric> RT-VH is configured to be an import or export RT of a particular set of
> Eric> VRFs.  But this text suggests that the V-hub has to know which of its
> Eric> import RTs is the RT-VH.

Fixed.

>    In handling such I-PMSI/S-PMSI/SA A-D route the V-hub follows the
>    [BGP-MVPN] procedures. Specifically, in contrast to Case 2 below, the
>    V-hub MUST NOT re-advertise this route.
> 
> Eric> Is the paragraph below meant to be a summary of what it means to
> Eric> "follow the [BGP-MVPN] procedures?   Or is it intended to specify some
> Eric> new procedure?  I think the former, but that should be clarified.  The
> Eric> [BGP-MVPN] procedure is that traffic traffic received from a P-tunnel
> Eric> does not get re-forwarded onto another P-tunnel, right?

The "procedures" in the above talk about handling I-PMSI/S-PMSI/SA A-D
*routes*. The text is fairly explicit on that. The above text does not
talk about handling traffic received in P-tunnels. The following
paragraph covers this.

>    The V-hub may forward traffic received on a P-tunnel bound to this I-
>    PMSI/S-PMSI A-D route to only the sites connected to that V-hub.  The
>    V-hub MUST NOT forward the traffic received on this P-tunnel to any
>    other V-hubs or V-spokes, including the V-spokes that import the VPN-
>    IP default route originated by the V-hub. Specifically, the V-hub
>    MUST NOT forward the traffic received on the P-tunnel advertised in
>    the received I-PMSI A-D route over the P-tunnel that the V-hub binds
>    to its Default I-PMSI A-D route.
> 
> 
> 7.8.2. Case 2
> 
>    This is the case where the received I-PMSI/S-PMSI/SA A-D route was
>    originated by either some other V-hub, or by a V-spoke. The I-PMSI/S-
>    PMSI/SA A-D route is imported by the V-hub (as well as by other V-
>    hubs), but not by any of the V-spokes that import the VPN-IP default
>    route originated by the V-hub.
> 
> Eric> How does the V-hub know that the A-D route is not imported by any of
> Eric> "the V-spokes that import the VPN-IP default route originated by the
> Eric> V-hub"?  This rule should be specified in terms of the RTs carried by
> Eric> the route, not in terms of what other PEs may or may not be doing.

See second paragraph in 7.8.2. I moved this paragraph up.

> Eric> BTW, I'd suggest using a term like "the V-spokes associated with the
> Eric> V-hub" instead of "the V-spokes that import the VPN-IP default
> Eric> route originated by the V-hub"

I added this clarification.

>    In this case the received I-PMSI/S-PMSI/SA A-D route does not carry
>    the RT that the receiving V-hub uses when advertising its VPN-IP
>    default route (RT-VH), although the route may carry more than one RT.
> 
> Eric> Again, I think there should be a clear statement of how a V-hub knows,
> Eric> by inspection of the RTs, when a received A-D route is a "case 2"
> Eric> route.

Done, as you requested.

>    In this case the V-hubs follows the procedures of [BGP-MVPN] with the
>    following additions.
> 
>    Once a V-hub accepts an I-PMSI A-D route, when the V-hub receives
>    data on the P-tunnel bound to that I-PMSI A-D route, the V-hub
>    follows procedures in [MVPN], [MVPN-BGP] to determine whether to
>    accept the data. If the data is accepted, then the V-hub further
>    forwards the data over the P-tunnel bound to the Default I-PMSI A-D
>    route originated by the V-hub.
> 
> Eric> This seems to rule out the possibility of a V-hub using an "S-PMSI
> Eric> only" model between a V-hub and its associated V-spokes, unless an
> Eric> "S-PMSI only" model is used uniformly throughout the MVPN.  Is this the
> Eric> intention?  If so, it should be stated explicitly.

Please note that (by definition) the "S-PMSI only" model assumes that 
no PEs use I-PMSI - quoting from rfc6625:

  ..."S-PMSI only" model, where no I-PMSIs are used at
  all for the given MVPN.

>From that definition it follows that for a given MVPN one can not have 
"S-PMSI only" model if at least some of the PEs of that VPN use I-PMSIs.

Thus if a V-hub accepts an I-PMSI A-D route (which is the scenario 
described in the paragraph), then we can not have the "S-PMSI only" 
model.

The fact that an "S-PMSI only" model has to be used uniformly
throughout the MVPN is due to the definition of this model made in
rfc6625.

However, your comment reminded me to cover the case of wildcard S-PMSIs.
The revised text does this. 

>    Note that in deciding whether to
>    forward the data over the P-tunnel bound to the Default I-PMSI A-D
>    route originated by the V-hub, the V-hub SHOULD take into account the
>    (multicast) state present in its MVPN-TIB that has been created as a
>    result receiving C-multicast routes from the V-spokes associated with
>    the V-hub.
> 
> Eric> Certainly the V-hub should take its multicast forwarding state into
> Eric> account when it does multicast forwarding ;-) Is something more
> Eric> specific intended here?

The revised text add a bit more.

>    Whenever a V-hub accepts an S-PMSI/SA A-D route, the V-hub, in
>    contrast to Case 1 above, MUST re-advertise this route to its V-
>    spokes. To accomplish this, the V-hub changes the RT carried by the
>    route to the RT that the V-hub uses when originating its VPN-IP
>    default route (RT-VH),
> 
> Eric> What if the route is carrying multiple RTs?  Which one gets changed?
> Eric> What happens to the others?   

The one that results in importing the route by the V-hub. Others
are unchanged. The revised text clarifies this.

>    and sets Next Hop to self. In addition, for S-
>    PMSI/SA A-D routes the V-hub changes the RD of the route to the RD
>    that the hub uses when originating its VPN-IP default route, and for
>    S-PMSI A-D routes the V-hub changes the Originating Router's IP
>    address in the MCAST-VPN NLRI of the route to its own IP address.
> 
>    Moreover, before re-advertising S-PMSI A-D routes to V-spokes, the V-
>    hub modifies the PMSI Tunnel attribute of these routes as appropriate
>    (e.g., by replacing the P-tunnel rooted at the originator of these
>    routes with a P-tunnel rooted at the V-hub).
> 
>    Whenever the V-hub receives data on the P-tunnel bound to the
>    received S-PMSI A-D route, the V-hub follows procedures of [MVPN],
>    [MVPN-BGP] to determine whether to accept the data. If the data is
>    accepted, then the V-hub further forwards it over the P-tunnel bound
>    to the S-PMSI A-D route that has been re-advertised by the V-hub.
> 
> Eric> At this point, there have been a lot of details, but still no
> Eric> statement of the what the data path for multicast traffic is supposed
> Eric> to be.  By reverse engineering the procedures, I conclude that the
> Eric> intention is to do the following (please correct me if I am wrong).
> Eric> Suppose V-spoke1 is associated with V-hub1, but not with V-hub2.
> Eric> Suppose V-spoke2 is associated with V-hub2, but not with V-hub1.
> Eric> Suppose there is a receiver, R, for (S,G) traffic, at a site attached
> Eric> to V-spoke1.  And suppose that S is at a site attached either to
> Eric> V-spoke2 or to V-hub2.  Now V-spoke1 will think the upstream PE for S
> Eric> is V-hub1.  So V-spoke1 joins (S,G) via V-hub1, and V-hub1 then joins
> Eric> (S,G) via V-spoke2 or V-hub2.  The data path of the (S,G) traffic that
> Eric> goes to R will be S-->V-hub2/V-spoke2-->V-hub1-->V-spoke1-->R.
> 
> Eric> I.e., it's like real multicast, with V-hub1 as an intermediate hop.
> 
> Eric> I understand why V-spoke1 does not communicate in the control plane
> Eric> with V-spoke2 or V-hub2.  But I don't really understand the point of
> Eric> using V-hub1 as a intermediate hop in the data plane.  For unicast
> Eric> traffic from a V-spoke, the associated V-hub needs to be an
> Eric> intermediate hop, because it needs to do an IP lookup in its VRF.  For
> Eric> multicast traffic, no IP lookup is done, and thus there is really no
> Eric> need to pass the data traffic through the associated V-hub.
> 

Please note that the procedures specified in the draft assume the
ability to perform sender-based RPF, as specified in 9.1.1 of
rfc6513. Given that, if one would follow what you outlined above,
could you point me to the specific text in 9.1.1 of rfc6513 that
would enable V-spoke1 to determine that from its own perspective
the UMH for (C-S, C-G) is V-hub1 ?

> Eric> On another topic, this section should explicitly address any issues
> Eric> that might arise if there are AS boundaries (or, if "seamless mpls' is
> Eric> being used, area boundaries) between the hubs and spokes, or between
> Eric> different hubs.  It shouldn't be left as an exercise to the reader to
> Eric> figure out whether the procedures work unchanged in those casees. 

As I mentioned before, the latest draft limits its scope to the case
where a given V-hub and all of the V-spokes associated with the
V-hub are in the same AS. 

>    If multiple S-PMSIs received by the V-hub have been aggregated into
>    the same P-tunnel, then the V-hub, prior to forwarding to the V-
>    spokes the data received on this P-tunnel may de-aggregate and then
>    re-aggregate (in a different way) this data using the state present
>    in its MVPN-TIB that has been created as a result of receiving C-
>    multicast routes from the V-spokes. Even if S-PMSIs received by the
>    V-hub each has its own P-tunnel, the V-hub, prior to forwarding to
>    the V-spokes the data received on these P-tunnels may aggregate these
>    S-PMSIs using the state present in its MVPN-TIB that has been created
>    as a result of receiving C-multicast routes from the V-spokes.
> 
> 
> 
> 7.9. Use of Ingress Replication with I-PMSI A-D routes
> 
>    The existing procedures for S-PMSI A-D routes are sufficient to
>    discard packets coming from a "wrong" PE for any type of P-tunnels,
>    including Ingress Replication.
> 
>    The following modifications to originating/receiving I-PMSI A-D
>    routes enable to discard packets coming from a "wrong" PE when
>    Ingress Replication is used for I-PMSI P-tunnels (for other types of
>    P-tunnels the procedures specified in [MVPN], [BGP-MVPN] are
>    sufficient).
> 
>    If Ingress Replication is used with I-PMSI A-D routes, then when a PE
>    advertises such routes, the Tunnel Type in the PMSI Tunnel attribute
>    is set to Ingress Replication; the Leaf Information Required flag is
>    set to 1; the the attribute carries no MPLS labels.
> 
> Eric> I believe this is incompatible with RFC 6514, which prohibits the
> Eric> setting of the Leaf Information Required flag in I-PMSI A-D routes.
> Eric> This violates RFC 6514, which prohibits the setting fo the LIR flag in
> Eric> I-PMSI A-D routes.
> 
> Eric> I do not see that it makes any sense to allow the LIR flag to be used
> Eric> in I-PMSI A-D routes for the V-hub/V-spoke architecture, while
> Eric> disallowing it otherwise.  I'm not even sure that's well-defined; how
> Eric> exactly will a PE know whether an I-PMSI A-D route with LIR set is
> Eric> legal in a particular scenario?
> 
> Eric> If it is considered desirable to allow the LIR flag to be used in
> Eric> I-PMSI A-D routes, a separate draft should be written, updating RFC
> Eric> 6514, that proposes to remove RFC 6514's restriction.  Presumably that
> Eric> draft would address interoperability issues with implementations that
> Eric> are compliant to RFC 6514 and disallow the LIR flag in I-PMSI A-D
> Eric> routes.  However, I have to say that I don't think this change is
> Eric> necessary, because the same effect can be obtained by using a wildcard
> Eric> S-PMSI.
> 
> Eric> In any case, I don't think a change like this should be hidden in "the
> Eric> fine print" of a draft that is on a different topic.

There is no "fine print" - the text is quite explicit. 

However, the revised text explicitly spells out the differences with 6514.

Note also, that there is nothing in the draft that precludes the use of 
a wildcard S-PMSI.

>    A PE that receives such I-PMSI A-D route MUST respond with a Leaf A-D
>    route.  The PMSI Tunnel attribute of that Leaf A-D route is
>    constructed as follows. The Tunnel Type is set to Ingress
>    Replication.  The Tunnel Identifier MUST carry a routable address of
>    the PE that originates the Leaf A-D route. The PMSI Tunnel attribute
>    MUST carry a downstream assigned MPLS label that is used to
>    demultiplex the traffic received over a unicast tunnel by the PE.
> 
> ...
> 
> 9. Further refinements
> 
>    In some cases a VPN customer may not want to rely solely on an (IP)
>    default route being advertised from a V-spoke to a CE, but may want
>    CEs to receive all the VPN routes (e.g., for the purpose of faster
>    detection of VPN connectivity failures, and activating some backup
>    connectivity).
> 
>    In this case one possible approach would be to install in the V-
>    spoke's data plane only the default route (following the Virtual Hub
>    and Spoke model, as described above), but keep all the VPN-IP routes
>    in the V-spoke's control plane (and thus being able to advertise
>    these routes from the V-spoke to the CEs).  Granted, this would not
>    change control plane resource consumption, but would (significantly)
>    reduce resource consumption on the data plane.
> 
> Eric> If the whole point of the V-hub/V-spoke architecture is to allow
> Eric> Spoke-PEs to maintain only a small set of routes, I don't quite see the
> Eric> point of having the spoke-PE maintain all the Internet routes.  

In the revised draft I made it clear that we are talking here about
reduced forwarding state.

Yakov.

---

From: Eric Rosen <[email-masked]>
Date: Wed, 21 Nov 2012 13:42:55 -0500

Yakov> I am in the process of addressing your comments. In this e-mail I'd
Yakov> like to focus on one particular one:

Yakov> Please note that the procedures specified in the draft assume
Yakov> the ability to perform sender-based RPF, as specified in 9.1.1
Yakov> of rfc6513.

The text of RFC6513 does not anticipate the use of a "virtual hub".  But
what's wrong with the following procedure:

- V-spoke1 selects V-hub1 as the upstream PE for (C-S,C-G).

- V-hub1 instructs V-spoke1 to receive (C-S,C-G) traffic from a
  unidirectional P-tunnel rooted at V-hub2.  (V-spoke1 follows this
  instruction because V-hub1 is its upstream PE for (C-S,C-G).)

- V-spoke1 drops any (C-S,C-G) traffic that did not arrive from V-hub2 over
  the specified P-tunnel.

Yakov> Furthermore, your outline above talks about S-PMSI A-D route. How
Yakov> would it work for I-PMSI A-D routes ?

Since V-spoke1 cannot determine the real upstream PE, V-spoke1 would
probably have to get a (C-S,C-G) S-PMSI A-D route from V-hub1 for each
(C-S,C-G) in which V-spoke1 has interest.  The PMSI Tunnel attribute would
identify the real upstream PE.  I think V-hub1 has the necessary state to
generate these, since it gets the C-multicast Joins its associated spokes.

Of course, there are trade-offs to be considered.

---

From: Yakov Rekhter <[email-masked]>
Date: Wed, 21 Nov 2012 11:46:28 -0800

Eric,

> - V-spoke1 drops any (C-S,C-G) traffic that did not arrive from V-hub2 over

You just confirmed my point that the existing procedures in 6513
are insufficient to support your alternative, and the draft is
very explicit in terms of its reliance on the 6513 procedures.

> generate these, since it gets the C-multicast Joins its associated spokes.

For one thing, this would rule out the use of I-PMSI between V-hub
and associated with it V-spokes, and would result in proliferation
of (C-S, C-G) S-PMSI routes.

Moreover, how would your alternative support DM (unsoliciated flooded
data) ?

How it would support (C-*, C-*) wildcard S-PMSI ?

Yakov.

---

From: Yakov Rekhter <[email-masked]>
Date: Wed, 21 Nov 2012 12:37:14 -0800

Eric,

> - V-spoke1 drops any (C-S,C-G) traffic that did not arrive from V-hub2 over

On more question, in addition to the questions I raised in the e-mail
I sent you earlier today:

How would your alternative support scenarios where the root of a
P-tunnel needs to know leaves of that tunnel (e.g., p2mp RSVP-TE
is used for P-tunnel, or the root performs aggregation using
upstream-assigned label)?  E.g., using your example above,
assume that V-hub2 originated an (C-S, C-G) S-PMSI A-D route with
Leaf Information Required. Please point me to the procedures in
rfc6513, rfc6514 that would result in a Leaf A-D route originated
by V-spoke1 to be received (and correctly processed) by V-hub2.

Yakov.

---

From: Eric Rosen <[email-masked]>
Date: Wed, 21 Nov 2012 16:08:55 -0500

So in the following case:

- source attached to V-hub2
- receiver attached to V-spoke1
- V-spoke1 associated with V-hub1

I was questioning whether it is really necessary to have a data path of
source-->V-hub2-->V-hub1-->V-spoke1-->receiver, or whether we could have the
more optimal data path of source-->V-hub2-->V-spoke1-->receiver

If I understand correctly, the answer is that in order to obtain the more
optimal path:

1. We would need a new procedure for determining which packets need to be
   discarded because they come from the "wrong place"; this would have to be
   tunnel-based rather than sender-based.  (A similar issue arises in
   extranet.) 

2. This procedure would require more BGP routes to be sent from a v-hub to
   its associated spokes, since in hub has to tell its associated spokes the
   tunnel to use for each (S,G).  (This makes it difficult to use I-PMSI or
   (*,*) S-PMSI A-D routes in the hub-to-associated-spoke control plane.)

While I agree that these points are valid, I don't see 1 as a real issue,
and 2 is a common tradeoff (increasing the optimality of the data plane
requires more control messages).  The issue of unsolicited flooded data is
really a sub-issue of 2.

However, in the context of WG LC, I think you have adequately answered the
question of why it makes the choice it does.

---

From: Eric Rosen <[email-masked]>
Date: Wed, 21 Nov 2012 16:15:01 -0500

> upstream-assigned label)?

And don't forget ingress replication ;-)

> E.g., using your example above,

Well, you've answered your own question.  On receiving an S-PMSI A-D route,
V-spoke1 would have to send a Leaf A-D route to the root of the tunnel (as
specified in the PMSI Tunnel attribute), rather than to the source of the
S-PMSI A-D route.  You are correct that this would require new procedures
that have to be specified.