Subject: [bess] Comments on draft-ietf-bess-evpn-overlay-04, wrt Inter-AS Option B

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Fri, 10 Jun 2016 20:03:00 +0000

I want to bring up an old discussion about the following:

   In summary, it can be seen that aliasing (and backup path)
   functionality should work as is for inter-AS option B without
   requiring any addition functionality in ASBRs or PEs. However, the
   mass-withdraw functionality falls back from per-ES mode to per-EVI
   mode for inter-AS option B - i.e., PEs receiving mass-withdraw route
   from the same AS use Ether A-D per ES route; whereas, PEs receiving
   mass-withdraw route from different AS use Ether A-D per EVI route.

There have been lot of discussions on this - offline and in Yokohama (https://www.ietf.org/proceedings/94/slides/slides-94-bess-1.pdf). The above text does not reflect the consensus among some of us and in particular does not reflect what was presented in Yokohama.

There are two issues with inter-as option B as currently specified in RFC 7432.

A minor issue is that mass-withdraw degrades from per ES to from per (ES, EVI) (with the clarification in this overlay draft), which would not exist if the main issue is resolved as presented in Yokohama.

The main one is the following requirement in RFC 7432:

   Note that the Ethernet A-D per EVI route may be received by a remote
   PE before it receives the set of Ethernet A-D per ES routes.
   Therefore, in order to handle corner cases and race conditions, the
   Ethernet A-D per EVI route MUST NOT be used for traffic forwarding by
   a remote PE until it also receives the associated set of Ethernet A-D
   per ES routes.

Basically, the per-EVI A-D routes cannot be used before the corresponding per-ES A-D routes are received and associated. Either that requirement must be removed, or there must be a way to associate the per-ES routes and per-EVI routes from the same PE.

There is an easy solution to the latter, which was presented in Yokohama (https://www.ietf.org/proceedings/94/slides/slides-94-bess-1.pdf). That does require a beefed up requirement: the routes from the same PE MUST have the same global admin field in the RDs. That should be reasonable, and RFC already uses RECOMMEND keyword:

7.9.  Route Distinguisher Assignment per MAC-VRF

   The Route Distinguisher (RD) MUST be set to the RD of the MAC-VRF
   that is advertising the NLRI.  An RD MUST be assigned for a given
   MAC-VRF on a PE.  This RD MUST be unique across all MAC-VRFs on a PE.
   It is RECOMMENDED to use the Type 1 RD [RFC4364].  The value field
   comprises an IP address of the PE (typically, the loopback address)
   followed by a number unique to the PE.

Notice the "RECOMMEND" in the above paragraph.

8.4.1.  Constructing Ethernet A-D per EVPN Instance Route
   ...
   The Route Distinguisher (RD) MUST be set per Section 7.9.

8.2.1.  Constructing Ethernet A-D per Ethernet Segment Route
   ...
   The Route Distinguisher (RD) MUST be a Type 1 RD [RFC4364].  The
   value field comprises an IP address of the PE (typically, the
   loopback address) followed by a number unique to the PE.

As long as 8.4.1 or 7.9 says Type 1 RD MUST be used, then all the problems are solved. While RFC 7432 did not use MUST, I doubt there is any implementation not using a Type 1 RD for the per-EVI route.

Jeffrey

---

From: "Ali Sajassi (sajassi)" <[email-masked]>
Date: Tue, 14 Jun 2016 03:00:47 +0000

Hi Jeffrey,

A few points: 

1) There have been lots of discussions on this topic but you were not in
attendance for some of them including the ones held at the last IETF in
Bones Aires. So, before jumping into a haste conclusion that there were
not consensus on section 10, please check with your own colleagues who
participated in those meetings.

2) Regarding the current text in section 10: this text is consistent with
the one that was checked for IETF at Yokohama and it is also consistent
with RFC 7432 operation. We still require Eth A-D per ES in order to
validate Eth A-d per EVI, the only difference is that the mass-withdraw
doesn¹t work (i.e., route validation still works).

3) Your so-called ³easy solution² doesn¹t work in all scenarios so there
is no point in documenting something that works sometimes :-) If you
recall several years ago, we went to great length to accommodate
multi-homing across multiple AS¹s and that¹s why we added originating
router¹s IP address to the ES route. The 4 byte field in RD type-1, is a
router ID (for both IPv4 and IPv6) and per RFC 6286, router-id is unique
within an AS only. Thus the suggested solution won¹t work when a CE is
dual-homed to two PEs in two different AS¹s.

We are currently separately documenting a proper solution based on a new
sub-TLV added to the attribute in idr-tunnel-encap draft to identify the
originating PE. A solution based on this attribute will work for all
scenarios. Now the question is what to do in short term. We can either go
with the text that it is in section 10.2 of the evpn-overlay draft, plus
go with the new draft, or just go with the new draft and remove the
suggested solution in section 10.2. I am open to both.

Cheers,
Ali 

On 6/10/16, 1:03 PM, "BESS on behalf of Jeffrey (Zhaohui) Zhang"
<[email-masked] on behalf of [email-masked]> wrote:

>There have been lot of discussions on this - offline and in Yokohama
>(https://www.ietf.org/proceedings/94/slides/slides-94-bess-1.pdf). The
>above text does not reflect the consensus among some of us and in
>particular does not reflect what was presented in Yokohama.
>
>A minor issue is that mass-withdraw degrades from per ES to from per (ES,
>EVI) (with the clarification in this overlay draft), which would not
>exist if the main issue is resolved as presented in Yokohama.
>
>Basically, the per-EVI A-D routes cannot be used before the corresponding
>per-ES A-D routes are received and associated. Either that requirement
>must be removed, or there must be a way to associate the per-ES routes
>and per-EVI routes from the same PE.
>
>There is an easy solution to the latter, which was presented in Yokohama
>(https://www.ietf.org/proceedings/94/slides/slides-94-bess-1.pdf). That
>does require a beefed up requirement: the routes from the same PE MUST
>have the same global admin field in the RDs. That should be reasonable,
>and RFC already uses RECOMMEND keyword:
>
>As long as 8.4.1 or 7.9 says Type 1 RD MUST be used, then all the
>problems are solved. While RFC 7432 did not use MUST, I doubt there is
>any implementation not using a Type 1 RD for the per-EVI route.
>
>Jeffrey
>
>_______________________________________________
>BESS mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bess

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Tue, 14 Jun 2016 09:44:39 +0000

Hi Ali,

> -----Original Message-----
> From: Ali Sajassi (sajassi) [mailto:[email-masked]]
> Sent: Monday, June 13, 2016 11:01 PM
> To: Jeffrey (Zhaohui) Zhang <[email-masked]>; [email-masked]
> Cc: Ali Sajassi (sajassi) <[email-masked]>
> Subject: Re: [bess] Comments on draft-ietf-bess-evpn-overlay-04, wrt
> Inter-AS Option B
> 
> 
> Hi Jeffrey,
> 
> A few points:
> 

Could the document clarify how the validation is done - how do you conclude that a per-ES route and a per-EVI route are related? If I understand it correctly, if the correlation can be done, then mass-withdraw works.

That is the key of the issue that is missing from the document.

> dual-homed to two PEs in two different AS¹s.

If those two PEs happen to have the same router-id, and they happen to choose the same local admin field for the RD, e.g. vlan id as section 7.9 of RFC 7432 says:

   The value field
   comprises an IP address of the PE (typically, the loopback address)
   followed by a number unique to the PE.  This number may be generated
   by the PE.  Or, in the Unique VLAN EVPN case, the low-order 12 bits
   may be the 12-bit VLAN ID, with the remaining high-order 4 bits set
   to 0.

How do you distinguish the two routes from the PEs? There is no way to guarantee it "always works" even if you don't use vlan-id.

> suggested solution in section 10.2. I am open to both.

Multi-homing across ASes (or any segmentation points) is complicated. I've spent quite some cycles on it - in particular split horizon becomes more complicated (I documented it in initial unpublished version of draft-zzhang-bess-evpn-bum-procedure-updates but then removed it). I would like to be included in the relevant discussions on the new document.

If we can say "multi-homing across ASes" is out of scope for now, then the overlay draft can certainly clarify inter-as optin B procedure (especially on how the correlation is done). Otherwise, I'd suggest to remove section 10.2. It is NOT specific to overlay anyway.

Jeffrey

---

From: "Ali Sajassi (sajassi)" <[email-masked]>
Date: Wed, 15 Jun 2016 05:59:21 +0000

Hi Jeffrey,

On 6/14/16, 2:44 AM, "Jeffrey (Zhaohui) Zhang" <[email-masked]> wrote:

>Could the document clarify how the validation is done - how do you
>conclude that a per-ES route and a per-EVI route are related? If I
>understand it correctly, if the correlation can be done, then
>mass-withdraw works.
>

It has been described in 4th paragraph of section 10.2.2:

"Now, when the AC between the PE2 and the CE fails and PE2 sends NLRI
   withdrawal for Ether A-D per ES route and this withdrawal gets
   propagated and received by the PE3, the BGP process in PE3 removes
   the corresponding BGP route; however, it doesn't remove the
   associated info (namely ESI and BGP next hop) from the L2 routing
   table (L2 RIB) because it still has the other Ether A-D per ES route
   (originated from PE1) with the same info. That is why the mass-
   withdraw mechanism does not work when doing DCI with inter-AS option
   B. However, as described previoulsy, the aliasing function works and
   so does "mass-withdraw per EVI" (which is associated with withdrawing
   the EVPN route associated with Aliasing - i.e., Ether A-D per EVI
   route).”

>If those two PEs happen to have the same router-id, and they happen to
>choose the same local admin field for the RD, e.g. vlan id as section 7.9
>of RFC 7432 says:
>
>   The value field
>How do you distinguish the two routes from the PEs? There is no way to
>guarantee it "always works" even if you don't use vlan-id.

In such scenarios, the ASBRs perform RD (and RT) translation; however, the
translated RD no longer needs to be of type-1.

>Multi-homing across ASes (or any segmentation points) is complicated.
>I've spent quite some cycles on it - in particular split horizon becomes
>more complicated (I documented it in initial unpublished version of
>draft-zzhang-bess-evpn-bum-procedure-updates but then removed it). I
>would like to be included in the relevant discussions on the new document.

Sure, we can definitely consider it.

Cheers,
Ali

>
>If we can say "multi-homing across ASes" is out of scope for now, then
>the overlay draft can certainly clarify inter-as optin B procedure
>(especially on how the correlation is done). Otherwise, I'd suggest to
>remove section 10.2. It is NOT specific to overlay anyway.
>
>Jeffrey
>
>> 
>> Cheers,
>> Ali
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> >There are two issues with inter-as option B as currently specified in
>>RFC
>> >7432.
>> >

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Wed, 15 Jun 2016 10:54:21 +0000

Hi Ali,

> Sent: Wednesday, June 15, 2016 1:59 AM
>    route)."

> >> >   a remote PE until it also receives the associated set of Ethernet A-D

For the aliasing part, how do you tell if a per-ES route and a per-EVI route are from the same PE? If you cannot tell, then the above validation cannot be done.

If we assume that they always have the same global admin field in the RDs, then the correlation can be done similar to how you associate the per-EVI route and mac routes as you clarified; but when I first proposed this solution when the option-B issue was brought up, people were saying that RFC 7432 does NOT require the per-ES route and per-EVI route have the same global admin field in the RDs, because:

- per-EVI route uses per-mac-vrf RD, which is RECOMMENDED to use type-1 (not MUST), with "typically the lookback address".
- per-ES route MUST uses a type-1 RD with "typically the loopback address".

Note that there is no explicit requirement/guarantee that the two will have the same global admin field in the RDs. If we put that explicit requirement in, then the problem is solved; and with that, mass-withdraw per ES also works.

Please note that it is not me who is splitting hair here. It is other people who were saying that RFC does not require they have the same global admin field.

So if we take this opportunity to tighten up the language, then we have an "easy solution" that always works (more below on that), and it removes hair splitting arguments from some people.

> translated RD no longer needs to be of type-1.

We discussed this before as well - as long as the RD translation ensures that routes with the same global admin field in the RDs continue to have the same global admin field that is different from those in routes from different PEs, then the "easy solution" still works. If it is not practical to have different admin field for routes from different PEs after translation, then the following requirement cannot be satisfied:

BTW - now that you say after translation the RD no longer needs to be of type-1, is there any reason to require type-1 at all, even when there is no inter-AS option B?

Jeffrey

---

From: John E Drake <[email-masked]>
Date: Wed, 15 Jun 2016 17:26:25 +0000

Jeffrey,

I don't think anyone would argue that the proper solution to per-ES mass withdraw is to include the originating router's address in Ethernet AD and MAC Advertisement routes and the only discussion is the best way to encode this information.

Your suggestion of having an ingress PE use the RD field to get the originating router's address has two issues:  

1)  RDs are known to be re-written by inter-provider ASBRs
2)  An IPv6 router address will not fit in an RD

Furthermore this is a fundamentally new behavior for an ingress PE and an implementation would need to be changed to support it;  no implementation of which I am aware does this today.

Given all of the above, the consensus is to include the originating router's address in the remote next-hop sub-TLV of the Encapsulation Attribute and modify egress PEs to include it and ingress PEs to use it.  This is a simple and incrementally deployable solution and in the meantime per-EVI mass withdraw can be used.

Yours Irrespectively,

John

> From: BESS [mailto:[email-masked]] On Behalf Of Jeffrey (Zhaohui) Zhang
> Sent: Wednesday, June 15, 2016 6:54 AM
> To: Ali Sajassi (sajassi); [email-masked]
> Subject: Re: [bess] Comments on draft-ietf-bess-evpn-overlay-04, wrt Inter-AS Option B
> 
> Hi Ali,
> 
> > >> Subject: Re: [bess] Comments on draft-ietf-bess-evpn-overlay-04,
> > >> wrt Inter-AS Option B
> > >>
> > >>
> > >> Hi Jeffrey,
> > >>
> > >> A few points:
> > >>
> > >> 1) There have been lots of discussions on this topic but you were
> > >> not
> > in
> > >> attendance for some of them including the ones held at the last
> > >> IETF in Bones Aires. So, before jumping into a haste conclusion
> > >> that there were not consensus on section 10, please check with your
> > >> own colleagues who participated in those meetings.
> > >>
> > >> 2) Regarding the current text in section 10: this text is
> > >>consistent with  the one that was checked for IETF at Yokohama and
> > >>it is also consistent  with RFC 7432 operation. We still require Eth
> > >>A-D per ES in order to  validate Eth A-d per EVI, the only
> > >>difference is that the mass-withdraw  doesn¹t work (i.e., route
> > >>validation still works).
> > >
> The above describes why only "mass-withdraw per EVI" works. The main problem is NOT
> with that. It's with the following in RFC 7432 as I mentioned initially in this thread:
> 
> For the aliasing part, how do you tell if a per-ES route and a per-EVI route are from the same
> PE? If you cannot tell, then the above validation cannot be done.
> 
> If we assume that they always have the same global admin field in the RDs, then the
> correlation can be done similar to how you associate the per-EVI route and mac routes as
> you clarified; but when I first proposed this solution when the option-B issue was brought
> up, people were saying that RFC 7432 does NOT require the per-ES route and per-EVI route
> have the same global admin field in the RDs, because:
> 
> Note that there is no explicit requirement/guarantee that the two will have the same global
> admin field in the RDs. If we put that explicit requirement in, then the problem is solved;
> and with that, mass-withdraw per ES also works.
> 
> Please note that it is not me who is splitting hair here. It is other people who were saying
> that RFC does not require they have the same global admin field.
> 
> So if we take this opportunity to tighten up the language, then we have an "easy solution"
> that always works (more below on that), and it removes hair splitting arguments from some
> people.
> 
> >
> > >
> > >>
> > >> is no point in documenting something that works sometimes :-) If
> > >> you recall several years ago, we went to great length to
> > >> accommodate multi-homing across multiple AS¹s and that¹s why we
> > >> added originating router¹s IP address to the ES route. The 4 byte
> > >> field in RD type-1, is
> > a
> > >> within an AS only. Thus the suggested solution won¹t work when a CE
> > >> is dual-homed to two PEs in two different AS¹s.
> > >
> > >If those two PEs happen to have the same router-id, and they happen
> > >to choose the same local admin field for the RD, e.g. vlan id as
> > >section 7.9 of RFC 7432 says:
> > >
> > >   The value field
> > >How do you distinguish the two routes from the PEs? There is no way
> > >to guarantee it "always works" even if you don't use vlan-id.
> >
> > In such scenarios, the ASBRs perform RD (and RT) translation; however,
> > the translated RD no longer needs to be of type-1.
> 
> We discussed this before as well - as long as the RD translation ensures that routes with the
> same global admin field in the RDs continue to have the same global admin field that is
> different from those in routes from different PEs, then the "easy solution" still works. If it is
> not practical to have different admin field for routes from different PEs after translation,
> then the following requirement cannot be satisfied:
> 
> BTW - now that you say after translation the RD no longer needs to be of type-1, is there
> any reason to require type-1 at all, even when there is no inter-AS option B?
> 
> Jeffrey
> 
> >
> > >
> > >>
> > >> We are currently separately documenting a proper solution based on
> > >> a
> > new
> > >> sub-TLV added to the attribute in idr-tunnel-encap draft to
> > >> identify
> > the
> > >> originating PE. A solution based on this attribute will work for
> > >>all  scenarios. Now the question is what to do in short term. We can
> > >>either go  with the text that it is in section 10.2 of the
> > >>evpn-overlay draft,
> > plus
> > >I've spent quite some cycles on it - in particular split horizon
> > >becomes more complicated (I documented it in initial unpublished
> > >version of draft-zzhang-bess-evpn-bum-procedure-updates but then
> > >removed it). I would like to be included in the relevant discussions on the new
> document.
> >
> > >If we can say "multi-homing across ASes" is out of scope for now,
> > >then the overlay draft can certainly clarify inter-as optin B
> > >procedure (especially on how the correlation is done). Otherwise, I'd
> > >suggest to remove section 10.2. It is NOT specific to overlay anyway.
> > >
> > >Jeffrey
> > >
> > >>
> > >> Cheers,
> > >> Ali
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> >(https://www.ietf.org/proceedings/94/slides/slides-94-bess-1.pdf).
> > >> >The above text does not reflect the consensus among some of us and in particular
> does not reflect what was presented in Yokohama.
> > >> >
> > >> >A minor issue is that mass-withdraw degrades from per ES to from
> > >> >per
> > >>(ES,
> > >> >EVI) (with the clarification in this overlay draft), which would
> > >> >not exist if the main issue is resolved as presented in Yokohama.
> > >> >
> > >> >   Ethernet A-D per EVI route MUST NOT be used for traffic
> > >> > forwarding
> > >>by
> > >> >   a remote PE until it also receives the associated set of
> > >> > Ethernet
> > >>A-D
> > >> >   per ES routes.
> > >> >
> > >> >per-ES A-D routes are received and associated. Either that
> > >> >requirement must be removed, or there must be a way to associate
> > >> >the per-ES routes and per-EVI routes from the same PE.
> > >> >
> > >> >does require a beefed up requirement: the routes from the same PE
> > >> >MUST have the same global admin field in the RDs. That should be
> > >> >reasonable, and RFC already uses RECOMMEND keyword:
> > >> >
> > >> >   MAC-VRF on a PE.  This RD MUST be unique across all MAC-VRFs on
> > >> > a
> > >>PE.
> > >> >problems are solved. While RFC 7432 did not use MUST, I doubt
> > >> >there is any implementation not using a Type 1 RD for the per-EVI route.
> > >> >
> > >> >Jeffrey
> > >> >

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Wed, 15 Jun 2016 20:24:51 +0000

Hi John,

I think there have been some disconnection that need to be cleared up first.

Section 10.2 of the evpn-overlay has the following two basic points about inter-as option B:

1. aliasing works as specified in RFC 7432 (plus clarification in the evpn-overlay draft)
2. mass-withdraw is only at EVI level not at ES level

The problem I have is with #1, not with #2. I have stated that in various discussions long time ago, in the first email of this thread, and repeated in my response to Ali in this thread.

To repeat, the problem with #1 is the following requirement in RFC 7432:

You can see that we need to correlate the per-ES route and per-EVI route in order to do aliasing. How that correlation is done needs to be specified/clarified. I suggested to use the global admin field of the RD.

If the current implementations do not use global admin field to do correlation, then inter-as option B does NOT work as is.

More below.

> From: John E Drake
> Sent: Wednesday, June 15, 2016 1:26 PM
> To: Jeffrey (Zhaohui) Zhang <[email-masked]>; Ali Sajassi (sajassi)
> <[email-masked]>; [email-masked]
> Cc: Ronald Bonica <[email-masked]>
> Subject: RE: [bess] Comments on draft-ietf-bess-evpn-overlay-04, wrt
> Inter-AS Option B
> 
> Jeffrey,
> 
> I don't think anyone would argue that the proper solution to per-ES mass
> withdraw is to include the originating router's address in Ethernet AD and
> MAC Advertisement routes and the only discussion is the best way to encode
> this information.

Please note that this is not about mass withdraw. Please see above.

> 
> Your suggestion of having an ingress PE use the RD field to get the
> originating router's address has two issues:

I am NOT saying to use the RD field to get the originating router's address. I am saying to use the global admin field of the RD to do the correlation. If a per-ES route's RD and a per-EVI route's RD has the same global admin field, then consider that the two routes are from the same PE.

That does require strong language to ensure per-ES route and per-EVI route from the same PE to have the same global admin field that is different from other PEs' RD's global admin field. That should be specified, and should be reasonable.

> 1)  RDs are known to be re-written by inter-provider ASBRs

That's OK; in my earlier discussions and my reply to Ali I said that as long as the rewriting keep the above requirement "per-ES route and per-EVI route from the same PE to have the same global admin field that is different from other PEs' RD's global admin field" it still works.

> 2)  An IPv6 router address will not fit in an RD

Again, the global admin field of the RD is used as opaque data, not as IP addresses.

> 
> Furthermore this is a fundamentally new behavior for an ingress PE and an
> implementation would need to be changed to support it;  no implementation
> of which I am aware does this today.

If an implementation follows the following in RFC 7432:

How will aliasing work "as is" in inter-as option B case?

While RFC 7432 does not explicit require "per-ES route and per-EVI route from the same PE to have the same global admin field that is different from other PEs' RD's global admin field" today, I would assert that existing implementations have effectually done that. As long as an ingress PE uses the global admin field to do the correlation, it should work fine. This does require new code, but I don't see other ways.

> 
> Given all of the above, the consensus is to include the originating
> router's address in the remote next-hop sub-TLV of the Encapsulation
> Attribute and modify egress PEs to include it and ingress PEs to use it.
> This is a simple and incrementally deployable solution and in the meantime
> per-EVI mass withdraw can be used.

This also requires implementation changes, doesn't it? But again, my main problem is NOT with mass withdraw. It's with aliasing.

Jeffrey

> forwarding by
> Ethernet A-D
> > - per-EVI route uses per-mac-vrf RD, which is RECOMMENDED to use type-1
> (not MUST),
> > with "typically the lookback address".
> > > That