Subject: [bess] John Scudder's Discuss on draft-ietf-bess-datacenter-gateway-10: (with DISCUSS and COMMENT)

From: John Scudder via Datatracker <[email-masked]>
Date: Thu, 13 May 2021 14:40:44 -0700

John Scudder has entered the following ballot position for
draft-ietf-bess-datacenter-gateway-10: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bess-datacenter-gateway/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I have several points I’d like to discuss, listed below from most general to
most specific.

1. There’s surprisingly little in this document that seems to be SR-specific
(and what there is, has some problems, see below). Is there some reason you
rule out interconnecting domains using other tunneling technologies? I ask this
question first because if the answer were to be “oh huh, we don’t need to make
this SR-specific after all” some of the other things I’m asking about might go
away.

2. There’s no discussion about what trust model you’re assuming. SR brings with
it its own assumed trust model, laid out in RFC 8402 as “SR operates within a
trusted domain” (whatever *that* means). On the one hand, given you’re tying
yourself to SR you presumably are tied to its trust model. On the other hand,
there are some tantalizing tidbits that suggest otherwise. I would be happier
if there were some explicit description of the trust model you’re presuming.
It’s hard to evaluate some aspects of the document without knowing if you’re
assuming the RFC 8402 closed domain model, or something else.

3. The use of the term “SR domain” in this document appears inconsistent with
its definition in RFC 8402. Here’s that definition, from §2:

   Segment Routing domain (SR domain): the set of nodes participating in
   the source-based routing model.  These nodes may be connected to the
   same physical infrastructure (e.g., a Service Provider's network).
   They may as well be remotely connected to each other (e.g., an
   enterprise VPN or an overlay).  If multiple protocol instances are
   deployed, the SR domain most commonly includes all of the protocol
   instances in a network.  However, some deployments may wish to
   subdivide the network into multiple SR domains, each of which
   includes one or more protocol instances.  It is expected that all
   nodes in an SR domain are managed by the same administrative entity.

And notably, later in 8402 §8 we are told that

   By default, SR operates within a trusted domain.  Traffic MUST be
   filtered at the domain boundaries.

Which specifically means, to take the MPLS instantiation of SR (§8.1):

   SR domain boundary routers MUST filter any external traffic destined
   to a label associated with a segment within the trusted domain.  This
   includes labels within the SRGB of the trusted domain, labels within
   the SRLB of the specific boundary router, and labels outside either
   of these blocks.  External traffic is any traffic received from an
   interface connected to a node outside the domain of trust.

More simply put, 8402 says you can’t send an SR packet from outside an SR
domain, into that domain. But your document is written in terms of a
multiplicity of SR domains, for example this in Section 1:

   Tunnel Encapsulation attribute.  The gateway in the ingress SR domain
   can now see all possible paths to X in the egress SR domain

Maybe a quick fix, assuming you really do subscribe to the RFC 8402 trust
model, is to invent, define, and use the term “SR subdomain” and deem all the
subdomains to comprise one SR domain, in the sense of RFC 8402 §2 — “They may
as well be remotely connected to each other (e.g., an enterprise VPN or an
overlay)” seems to describe your situation pretty well.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

1. Abstract

   This document defines a mechanism using the BGP Tunnel Encapsulation
   attribute to allow each gateway router to advertise the routes to the
   prefixes in the Segment Routing domains to which it provides access,
   and also to advertise on behalf of each other gateway to the same
   Segment Routing domain.

The last clause has no object. To advertise WHAT? Possibly (?) it could be
rewritten “... provides access, including advertising them on behalf of...”

2. Section 1

   against connection of device failure.

“Of” -> “or”

3. Section 3

   o  A route target ([RFC4360]) is attached to each GW's auto-discovery
      route and has its value set to the SR domain identifier.

Insert “(defined below)” after “auto-discovery route”, please?

   The auto-discovery route that each GW advertises consists of the
   following:

The use of the definite article implies that each GW can advertise one, and
only one, auto-discovery route. Is this true?

4. Section 5

   When a packet destined for prefix X is sent on an SR TE path to a GW
   for the SR domain containing X (that is, the packet is sent in the
   Ingress Domain on an SR TE path that describes the path including
   within the Egress Domain), it needs to carry the receiving GW's label

I can’t understand the parenthetical, in particular “the path including within
the Egress Domain”.

Also, do you really mean “label”, or do you mean “SID”? I don’t think you
scoped this to only SR-MPLS, did you? Although reading on within §5 you talk
about the “label stack”, so it does appear you’re MPLS specific — probably this
should be said up front, in that case? The title should really be “… for
SR-MPLS Enabled Domain Interconnection”?

5. Section 6

   If the GWs for a given SR domain are configured to allow remote GWs
   to send them a packet in that SR domain's native encapsulation, then

This is forbidden by RFC 8402, see my discuss comment #3. Maybe this is just an
easy terminology fix, or maybe not.

6. Section 8

   All of the issues in the list above could cause disruption to domain
   interconnection, but are not new protocol vulnerabilities so much as
   new exposures of information that SHOULD be protected against using
   existing protocol mechanisms.  Furthermore, it is a general

What are the existing BGP protocol mechanisms that could be used to protect
against exposure of information? BGP itself doesn’t have any confidentiality
features nor do most of its common transports. Maybe you mean something
different, but if so that’s not clear to me.

   system.  It should be noted that BGP peerings are not discovered, but
   always arise from explicit configuration.

This is true at present, but IDR has work in progress on autodiscovery. (Same
comment applies with respect to Section 9.)

7. Section 9.1

   consideration.  When using the mechanisms defined in this document,
   the operator should consider carefully the effects of filtering
   routes.  In some cases this may be desirable, and in others it could
   limit the effectiveness of the procedures.

I believe the only use of route targets in this document is for the
autodiscovery routes.  If RTC were in use, through its normal operation the
gateways would exchange autodiscovery routes exactly as this specification
needs them to. So your cryptic warning above leaves me wondering, what are the
cases in which RTC impedes the function of the specification?

8. General

The autodiscovery mechanism is clear as far as it goes, but I think not all
failure modes are addressed. In particular, if there’s partial connectivity
within a domain, I think long-term black holing can ensue. Consider this case:
GW1 and GW2 are gateways in domain A. GW3 is a gateway in domain B. GW1 and GW2
discover one another and advertise one another’s encapsulation information
accordingly, when advertising a route to prefix X. However, there’s a problem
within GW1 and GW2’s domain, such that GW1 can reach X, but GW2 can’t. Even
though GW2 may know it can’t reach X, and indeed GW2 isn’t advertising X, GW1
is still advertising GW2 as a viable gateway to reach X, and GW3 may well route
traffic for X via GW2.

Admittedly, having partial connectivity within a domain as I’ve described is a
broken situation to begin with, but stuff happens, and your spec would make
matters worse. It might be worth acknowledging this issue somewhere in the
document?

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 14 May 2021 16:39:53 +0100

Hi John,

I'm currently constructing a reply to your points. Extensive review deserves extensive answers. May take another day or two.

Cheers,
Adrian

-----Original Message-----
From: John Scudder via Datatracker <[email-masked]> 
Sent: 13 May 2021 22:41
To: The IESG <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; Matthew Bocci <[email-masked]>; [email-masked]
Subject: John Scudder's Discuss on draft-ietf-bess-datacenter-gateway-10: (with DISCUSS and COMMENT)

1. Abstract

2. Section 1

“Of” -> “or”

3. Section 3

4. Section 5

5. Section 6

6. Section 8

7. Section 9.1

8. General

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 14 May 2021 18:04:37 +0100

Hi John,

Thanks for the careful review.

> DISCUSS:
>
> I have several points I’d like to discuss, listed below from most
> general to most specific.
>

I'm sorry this isn't clear, but the use of other tunnelling technologies is very much in scope. As the Introduction says:

   The
   various ASes that provide connectivity between the Ingress and Egress
   Domains could each be constructed differently and use different
   technologies such as IP, MPLS with global table routing native BGP to
   the edge, MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN.

SR is used to identify the tunnels and provide end-to-end SR paths because the ingress and egress domains are SR domains, and the objective is to provide an end-to-end SR path.

So we are not "making this SR aware" so much as enabling "SR-over-foo" using SIDs to identify the path segments that are tunnels.

I don't know how to make this clearer except maybe using some red paint. We would write...

   The
   various ASes that provide connectivity between the Ingress and Egress
   Domains could each be constructed differently and use different
   technologies such as IP, MPLS with global table routing native BGP to
   the edge, MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN.  That is, the
   Ingress and Egress SR Domains can be connected by tunnels across a
   variety of technologies.  This document describes how SR identifiers
   (SIDs) are use to identify the paths between the Ingress and Egress
   and the techniques in this document apply to routes of all AFI/SAFIs.

> 2. There’s no discussion about what trust model you’re assuming. SR
> brings with it its own assumed trust model, laid out in RFC 8402 as “SR
> operates within a trusted domain” (whatever *that* means). On the one
> hand, given you’re tying yourself to SR you presumably are tied to its trust
> model. On the other hand, there are some tantalizing tidbits that suggest
> otherwise. I would be happier if there were some explicit description of
> the trust model you’re presuming. It’s hard to evaluate some aspects of
> the document without knowing if you’re assuming the RFC 8402 closed
> domain model, or something else.

I believe that the term "SR domain" in 8402 is basically defined as "a set of nodes that support SR". 
The description in (the ever-so-skimpy section 8 of 8402) says:

What does "by default" mean in that context? 

I think there are two things to think about:

1. Forwarding plane trust model. Can packets get into the SR system? The answer to that remains, "No, because traffic MUST be filtered at the domain boundaries." This requires that the domain boundary is the interface between an SR-capable node, and a non-SR node. In this document all GWs and ASBRs are part of the SR domain connected by tunnels across the transit ASes (although the nodes in the transit ASes are not part of that domain). I dare say that draft-farrel-spring-sr-domain-interconnect explains this better through examples, but the chairs of SPRING told us that that draft had no chance of progressing.

2. Control plane trust model. What is the trust model in the BGP system? I'm pretty sure that Section 8 of our draft is adequate for this discussion,.

> overlay)” seems to describe your situation pretty well.

We completely agree with the 8402 meta-definition of SR domain (and I used it to answer your previous point).

The confusion appears to arise purely from the terms "ingress SR domain" and "egress SR domain" which is our bad choice of words. "Site" would be a better word and I will scrub the document to use that. "Subdomain" seems to exacerbate the already-dubious word "domain."

> rewritten “... provides access, including advertising them on behalf of...”

Yeah, OK.

>   against connection of device failure.

Ack

> Insert “(defined below)” after “auto-discovery route”, please?

OK

> only one, auto-discovery route. Is this true?

Indeed, just one (with potentially multiple tunnel encaps). We can make that explicit unless you can find a reason why advertising more than one would be beneficial.

> I can’t understand the parenthetical, in particular “the path including within

"describes the whole path including those parts that are within the Egress Site"

> SR-MPLS Enabled Domain Interconnection”?

Ouch! Yes, that slipped through. It's SIDs all the way down.

> easy terminology fix, or maybe not.

Yes, this is the "When is a domain not a domain?" question.
The answer is either "When it's a jar" or "We mean 'site'."

> different, but if so that’s not clear to me.

I don't think we intended (or said) "BGP protocol mechanisms." But other protocol mechanisms *do* exist to protect BGP exchanges between peers. TLS or TCP-AO spring to mind.

The point here, is that it is not the job of this document to solve BGP security issues.

> comment applies with respect to Section 9.)

Weeeeeeell, I think that that IDR work will need to provide adequate security.

> cases in which RTC impedes the function of the specification?

That is really what we intended you to wonder about. I suppose we are saying, "We are worried it might have an effect, but we can't put our finger on it."

> matters worse. It might be worth acknowledging this issue somewhere in the

Agree with you that "stuff happens." I think that what you have described is a window not a permanent situation.
When GW2 knows it can't reach X any more, it will stop advertising X, and GW1 will receive that and will update what it advertises on behalf of GW2.
Further, if GW1 can no longer receive advertisements from GW2 then it will stop advertising on behalf of GW2.

Cheers,
Adrian

---

From: John Scudder <[email-masked]>
Date: Fri, 14 May 2021 20:12:05 +0000

Hi Adrian,

Thanks for your reply. Pressed for time at the moment but one partial response:

On May 14, 2021, at 1:04 PM, Adrian Farrel <[email-masked]<mailto:[email-masked]>> wrote:

Ah, perhaps I have badly misunderstood the way this works. I had thought it went something like this:

- GW1 knows it can reach GW2 because of GW2’s auto discovery route
- GW1 knows the set S of internal prefixes it can reach
- GW1 advertises each prefix from S with both GW1 and GW2 in the tunnel attribute

In the description above, there’s no notion of GW2 telling GW1 what internal prefixes GW2 can reach, or GW1 caring.  Now I suppose you are telling me that it goes:

- GW1 knows it can reach GW2 because of GW2’s auto discovery route
- GW1 knows the full set of prefixes GW2 can reach. _How does it know this?_
- GW1 constructs each advertisement listing only the correct set of gateways in the tunnel attribute

The key question is the one I’ve highlighted: how does GW1 come to know GW2’s internally-reachable prefixes? I didn’t notice any of this in the spec. Maybe it was just my sloppy reading, I’ll look again.

Yes, that’s understood, but I was positing a case where just because GW1 can reach GW2 stably, and just because GW1 can reach X stably, it does not imply GW2 can reach X.

—John

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 14 May 2021 22:36:39 +0200

Hi John,

The way I understood this is intending to work in practice is simply to
create IBGP session between GW1 & GW2,

If we have this IBGP session then there are two cases:

* we receive route to X from peer GW so we know peer GW can reach X hence
it is safe to advertise X with both GWs as NHs
* we send route to X to peer GW so we know that peer GW can reach X (at
least via sender's GW) hence it is again ok to  advertise X with both GWs
as NHs

Seems like this logic can solve your question ...

But good catch :)

Cheers,
Robert

On Fri, May 14, 2021 at 10:12 PM John Scudder <jgs=
[email-masked]> wrote:

> On May 14, 2021, at 1:04 PM, Adrian Farrel <[email-masked]> wrote:
>
> When GW2 knows it can't reach X any more, it will stop advertising X, and
> GW1 will receive that and will update what it advertises on behalf of GW2.
>
>
> In the description above, there’s no notion of GW2 telling GW1 what
> internal prefixes GW2 can reach, or GW1 caring.  Now I suppose you are
> telling me that it goes:
>
> The key question is the one I’ve highlighted: how does GW1 come to know
> GW2’s internally-reachable prefixes? I didn’t notice any of this in the
> spec. Maybe it was just my sloppy reading, I’ll look again.
>
> Yes, that’s understood, but I was positing a case where just because GW1
> can reach GW2 stably, and just because GW1 can reach X stably, it does not
> imply GW2 can reach X.
>
> —John
> _______________________________________________
> BESS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bess
>

---

From: John Scudder <[email-masked]>
Date: Fri, 14 May 2021 21:24:45 +0000

Having re-read Section 3 carefully (and skimmed the rest) I still think what the document says (as opposed to what’s in the authors’ heads?) is the first description I give below. Let me know if you want me to walk through my reasoning in detail with reference to the document.

—John

On May 14, 2021, at 4:12 PM, John Scudder <[email-masked]> wrote:

 Hi Adrian,

—John

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 14 May 2021 19:02:47 -0400

Hi Adrian

I believe what John is describing is a valid failure scenario where one of
the GWs is no longer a valid gateway because it’s eBGP peering to core
domain is down, however the routing underlay is stable between the GWs
within the DC site. We are assuming the GWs at the site run an IGP to
advertise next hop attribute and maybe also have next hop self configured
between then for iBGP.  So the GW that had eBGP peer to core is able to
send the auto discovery loopback prefix tunnel sub TLV for both GWs.
That’s the problem.

So that would cause the black hole of traffic between sites for the GW that
has its eBGP link to the core down.

The other question asked was eve set of internal prefixes how are they
advertised and is that just over the native AFI SAFI iBGP peering between
the GWs.

So GW1 that is up advertises via iBGP the set of internal prefixes learned
from the other domain.

Kind Regards

Gyan

On Fri, May 14, 2021 at 5:25 PM John Scudder <jgs=
[email-masked]> wrote:

> Having re-read Section 3 carefully (and skimmed the rest) I still think
> what the document says (as opposed to what’s in the authors’ heads?) is the
> first description I give below. Let me know if you want me to walk through
> my reasoning in detail with reference to the document.
>
> —John
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 14 May 2021 22:45:21 -0400

Hi Adrian

I may have missed this in the draft but the solution for this failover
scenario is if each GW can only advertise itself, which I think that is
stated in section 3 then GW1 can only advertise itself via tunnel
encapsulation attribute and not GW2 as GW2 can only advertise itself when
it’s eBGP tie to the core comes Up.  Problem solved.

Kind Regards

Gyan

On Fri, May 14, 2021 at 7:02 PM Gyan Mishra <[email-masked]> wrote:

> So that would cause the black hole of traffic between sites for the GW
> that has its eBGP link to the core down.
>

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 15 May 2021 00:35:44 -0400

Adrian & Authors please correct me if I misspeak the way I read the draft.

I did not see in the draft stating explicitly how the internal DC GW routes
are advertised which I believe implicitly done via standard BGP AFI / SAFI
route propagation natively over the SR domain.  So for example if the
internal GW routers are SAFI 1 unicast they are propagated natively global
table routed as SAFI 1 inter domain from ingress to egress DC GW.

For VPN overlay SAFI 128 129 routes are propagated natively as SAFI 128 129
over the SR core.

So all GW internal routes AFI / SAFI are supported meaning SAFI  1, 128,
129.

Kind Regards

Gyan

On Fri, May 14, 2021 at 10:45 PM Gyan Mishra <[email-masked]> wrote:

>> I believe what John is describing is a valid failure scenario where one
>> of the GWs is no longer a valid gateway because it’s eBGP peering to core
>> So GW1 that is up advertises via iBGP the set of internal prefixes
>> learned from the other domain.
>>
>> Kind Regards
>>
>> Gyan
>>
>>> Agree with you that "stuff happens." I think that what you have
>>> described is a window not a permanent situation.
>>> When GW2 knows it can't reach X any more, it will stop advertising X,
>>> and GW1 will receive that and will update what it advertises on behalf of
>>> GW2.
>>>
>>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 16 May 2021 01:32:47 -0400

Section 3 verbiage below describes the
re-advertisement of current set of GWs due to GW being added or deleted.

So the blackhole John mentioned due to a GW being disconnected from
backbone should not occur.

   As described in Section 1
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-datacenter-gateway-10#section-1>,
each

   GW will include a Tunnel

   Encapsulation attribute with the GW encapsulation information for
   each of the SR domain's active GWs (including itself) in every route
   advertised externally to that SR domain.  As the current set of
   active GWs changes (due to the addition of a new GW or the failure/
   removal of an existing GW) each externally advertised route will be
   re-advertised with a new Tunnel Encapsulation attribute which
   reflects current set of active GWs.

   If a gateway becomes disconnected from the backbone network, or if
   the SR domain operator decides to terminate the gateway's activity,
   it withdraws the advertisements described above.  This means that
   remote gateways at other sites will stop seeing advertisements from
   this gateway.

I support this very important draft that provides a critical solution
for remote site GW auto-discovery over a Multi-AS transit SR / MPLS
domain.

For those whom have not read normative reference below I suggest
reading as it helps provide context to the problem statement and
solution.

https://datatracker.ietf.org/doc/html/draft-farrel-spring-sr-domain-interconnect-05

Kind Regards

Gyan

On Sat, May 15, 2021 at 12:35 AM Gyan Mishra <[email-masked]> wrote:

> I did not see in the draft stating explicitly how the internal DC GW
> routes are advertised which I believe implicitly done via standard BGP AFI
> / SAFI route propagation natively over the SR domain.  So for example if
> the internal GW routers are SAFI 1 unicast they are propagated natively
> global table routed as SAFI 1 inter domain from ingress to egress DC GW.
>
> For VPN overlay SAFI 128 129 routes are propagated natively as SAFI 128
> 129 over the SR core.
>
>>>> Yes, that’s understood, but I was positing a case where just because
>>>> GW1 can reach GW2 stably, and just because GW1 can reach X stably, it does
>>>> not imply GW2 can reach X.
>>>>
>>>> —John
>>>>

*Gyan Mishra*

*M 301 502-1347*

---

From: "Adrian Farrel" <[email-masked]>
Date: Sun, 16 May 2021 12:25:04 +0100

Hi John,

Trying to dismantle this…

We are saying that a site is integral. You are is asking : what happens if a site becomes partitioned so that some prefixes are accessible through one GW and some through another.

Consider a site with a set of prefixes S

Consider two GWs: GW1 and GW2

Initially GW1 and GW2 discover each other.

So GW1 advertises reachability to S, and by the way GW2 exists

GW2 advertises reachability to S, and by the way GW1 exists

Now the site becomes partitioned so that GW1 can reach S1 and GW2 can reach S2. (S = S1 U S2,  S1 n S2 = E)

You ask:

1.	What happens to packets for S2 arriving at GW1?
2.	What is the remedy in the protocol?

My answer to 1. is that the packets will be black-holed either at GW1 or inside the site.

My observation is that:

a.	GW1 cannot reach GW2 inside the site. If it could, then S2 would be reachable via GW1
b.	It is contrary to BCP38 for GW1 to forward a packet back into the external AS to be routed to GW2

My answer to 2. is that when the site becomes partitioned:

*	GW1 will stop advertising the whole of S and will fall back to advertising just S1
*	GW2 will stop advertising the whole of S and will fall back to advertising just S2
*	Initially, GW1 and GW2 will still advertise each other’s existence, but will “soon” un-auto-discover each other

At this point the site is effectively two sites that use the same site identifier.

How quickly this takes place depends on precisely what the failure case is, how fast the failure detection is done, and how fast BGP converges.  

*Perhaps* there is a wrinkle *if* the autodetection advertisements are sent external to the site. In this case, GW1 would continue to discover GW2 and so would readvertise it (and vice versa). This would continue to lead to the broken condition you noted. I think we assumed that the peering between GW1 and GW2 would be internal to the site (because otherwise it would constitute traffic leaving the site and re-entering it (breaking BCP38 again). If it would help, we could make this point clear by saying that the peering between GW1 and GW2 must be within the site.

Cheers,

Adrian

From: John Scudder <[email-masked]> 
Sent: 14 May 2021 22:25
To: Adrian Farrel <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Matthew Bocci <[email-masked]>
Subject: Re: John Scudder's Discuss on draft-ietf-bess-datacenter-gateway-10: (with DISCUSS and COMMENT)

—John

 Hi Adrian, 

On May 14, 2021, at 1:04 PM, Adrian Farrel <[email-masked] <mailto:[email-masked]> > wrote:

—John

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 17 May 2021 13:50:25 -0400

Adrian

I am wrapping up the Gen-ART review update.

The normative draft helped tremendously in understanding the problem and
solution.  Please add to the beginning of the introduction in your next
update.

https://datatracker.ietf.org/doc/html/draft-farrel-spring-sr-domain-interconnect-05#section-9

I have a question related to the partitioning scenario.

Comments in-line

On Sun, May 16, 2021 at 7:25 AM Adrian Farrel <[email-masked]> wrote:

> We are saying that a site is integral. You are is asking : what happens if
> a site becomes partitioned so that some prefixes are accessible through one
> GW and some through another.
>
>
>

    Gyan> Ack

>  Gyan> Ack
>    1. GW1 cannot reach GW2 inside the site. If it could, then S2 would be
>    reachable via GW1
>    2. It is contrary to BCP38 for GW1 to forward a packet back into the
>    external AS to be routed to GW2
>
>  Gyan> How would “b” be possible as loop avoidance would drop the packet
> sent from GW1 external to loop back in GW2 being part of the same AS would
> drop the packets so they would not be able to re-discover each other via
> external.
>
>    - GW1 will stop advertising the whole of S and will fall back to
>    advertising just S1
>    - GW2 will stop advertising the whole of S and will fall back to
>    advertising just S2
>    - Initially, GW1 and GW2 will still advertise each other’s existence,
>    but will “soon” un-auto-discover each other
>
>
>    -
>
>  Gyan> Ack
> How quickly this takes place depends on precisely what the failure case
> is, how fast the failure detection is done, and how fast BGP converges.
>
>  Gyan> Ack
>

**Perhaps** there is a wrinkle **if** the autodetection advertisements are
> sent external to the site. In this case, GW1 would continue to discover GW2
> and so would readvertise it (and vice versa).
>
Gyan> As I stated above  “b” would not be possible so the continuing
rediscovery would not occur and GW1 and GW2 would as you stated effectively
act as two sites.

This would continue to lead to the broken condition you noted.

Gyan> As stated due to as-path loop avoidance the broken condition would
not occur.

I think we assumed that the peering between GW1 and GW2 would be internal
> to the site (because otherwise it would constitute traffic leaving the site
> and re-entering it (breaking BCP38 again). If it would help, we could make
> this point clear by saying that the peering between GW1 and GW2 must be
> within the site.
>
Gyan>I agree we should add explicit verbiage that iBGP per BCP38 to prevent
site partitioning as well as required for both GW1 and GW2 to be part of
the same AS and be able to provide failover and back each other Up in case
of a failure.

Gyan> I believe John mentioned this slightly different scenario but it got
lost in the thread and I tried to answer early on but I  would like to get
your take on the behavior.  As this is critical component to complete my
Gen-Art review.

So if GW2 connection to external was down but GW1 still has its connection
to external.  GW2 would auto discover GW1 over iBGP and GW2 would advertise
both GW1 and GW2 as reachable gateways.  However GW2 has its external peer
down.  So if GW1 continues to advertised GW2 as we stated GW1 will auto
discover  GW2 over iBGP.

So now for any sites trying to reach the Data Center AS that GW1 and GW2
are part of using GW2 to get to S1 and S2 would be black hole.  How do we
remedy this situation.

>
> Cheers,
>
> Adrian
>
>
>
> *From:* John Scudder <[email-masked]>
> *Sent:* 14 May 2021 22:25
> *To:* Adrian Farrel <[email-masked]>
> *Cc:* The IESG <[email-masked]>;
> [email-masked]; [email-masked];
> [email-masked]; Matthew Bocci <[email-masked]>
> *Subject:* Re: John Scudder's Discuss on
> draft-ietf-bess-datacenter-gateway-10: (with DISCUSS and COMMENT)
>
>
>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: John Scudder <[email-masked]>
Date: Mon, 17 May 2021 19:32:04 +0000

Hi Adrian,

Comments in line.

> On May 16, 2021, at 7:25 AM, Adrian Farrel <[email-masked]> wrote:
> 
> 
> Hi John,
>  
> We are saying that a site is integral.

I don’t think I saw any place in the draft that states that assumption.

> You are is asking : what happens if a site becomes partitioned so that some prefixes are accessible through one GW and some through another.

Actually that wasn’t my scenario, I see I must have expressed it poorly. I’ll come back to that, but first, your own scenario.
 
> 	• What happens to packets for S2 arriving at GW1?
> 	• What is the remedy in the protocol?
>  
> 	• GW1 cannot reach GW2 inside the site. If it could, then S2 would be reachable via GW1
> 	• It is contrary to BCP38 for GW1 to forward a packet back into the external AS to be routed to GW2
>  
> 	• GW1 will stop advertising the whole of S and will fall back to advertising just S1
> 	• GW2 will stop advertising the whole of S and will fall back to advertising just S2

This, too, is unstated in the draft, and it’s not like it will happen for free by natural operation of every router. But in the scenario you’ve picked that seems OK since this practice is needed for a multi homed site regardless of whether your draft is in use or not — if the site partitions, either you deaggregate, or you black hole. (“Deaggregate” might be actual deaggregation, or simply advertising only the subset of individual prefixes that remain reachable, depending on the address allocation scheme in use, of course.)

> 	• Initially, GW1 and GW2 will still advertise each other’s existence, but will “soon” un-auto-discover each other

Seems reasonable. However, it wasn’t the problem I was posing. Since that seems to have gotten lost in the quote-and-snip chain, here it is again:

"The autodiscovery mechanism is clear as far as it goes, but I think not all failure modes are addressed. In particular, if there’s partial connectivity within a domain, I think long-term black holing can ensue. Consider this case: GW1 and GW2 are gateways in domain A. GW3 is a gateway in domain B. GW1 and GW2 discover one another and advertise one another’s encapsulation information accordingly, when advertising a route to prefix X. However, there’s a problem within GW1 and GW2’s domain, such that GW1 can reach X, but GW2 can’t. Even though GW2 may know it can’t reach X, and indeed GW2 isn’t advertising X, GW1 is still advertising GW2 as a viable gateway to reach X, and GW3 may well route traffic for X via GW2. 

The key difference between the question you answered, and the one I asked, is that you answer what happens in the case of a complete partition, and I ask what happens in the case of inconsistent routing within the site. When you worked through your scenario, you included this:

> 	• GW1 cannot reach GW2 inside the site. If it could, then S2 would be reachable via GW1

That isn’t the case in my scenario. GW1 can reach GW2, GW1 cannot reach S2. I think the assumption you stated up top (“the site is integral”) answers this question too: this case isn’t covered since it violates the assumption. I think that’s OK, it’s a case I’d call silly if I hadn’t seen it happen, but it’s still low-probability. The question was only, is it worth stating the assumption and what is/isn’t covered?

"Admittedly, having partial connectivity within a domain as I’ve described is a broken situation to begin with, but stuff happens, and your spec would make matters worse. It might be worth acknowledging this issue somewhere in the document?”

I hope this is clearer now.

Thanks,

—John

---

From: John Scudder <[email-masked]>
Date: Mon, 17 May 2021 19:55:41 +0000

Hi Gyan, 

> On May 17, 2021, at 1:50 PM, Gyan Mishra <[email-masked]> wrote:
> 
> So if GW2 connection to external was down but GW1 still has its connection to external.  GW2 would auto discover GW1 over iBGP and GW2 would advertise both GW1 and GW2 as reachable gateways.  However GW2 has its external peer down.  So if GW1 continues to advertised GW2 as we stated GW1 will auto discover  GW2 over iBGP.  

Isn’t this scenario covered? From §3:

So when GW2’s external peering goes down, GW2 withdraws its auto discovery route, and therefore GW1 re-advertises its routes externally without GW2 listed in the tunnel attribute.

I will say that reviewing the above-quoted text — which seems tailor-made for a “MUST withdraw” — made me notice that the draft makes only sporadic and desultory use of RFC2119 keywords. In fact there are so few used, that it seems like it might be better to scrub those two SHOULD and two MUST out and remove the 2119 citation.

—John

---

From: Alvaro Retana <[email-masked]>
Date: Mon, 17 May 2021 13:11:48 -0700

On May 14, 2021 at 1:04:53 PM, Adrian Farrel wrote:

Hi!

I share some of John's concerns -- quick comment on the first one.

...
> > 1. There’s surprisingly little in this document that seems to be
> > SR-specific (and what there is, has some problems, see below). Is there
> > some reason you rule out interconnecting domains using other tunneling
> > technologies? I ask this question first because if the answer were to be
> > “oh huh, we don’t need to make this SR-specific after all” some of the
> > other things I’m asking about might go away.
>
> I'm sorry this isn't clear, but the use of other tunnelling technologies is
> very much in scope. As the Introduction says:
>
> The various ASes that provide connectivity between the Ingress and Egress
> Domains could each be constructed differently and use different technologies
> such as IP, MPLS with global table routing native BGP to the edge, MPLS IP
> VPN, SR-MPLS IP VPN, or SRv6 IP VPN.
>
> SR is used to identify the tunnels and provide end-to-end SR paths because the
> ingress and egress domains are SR domains, and the objective is to provide an
> end-to-end SR path.
>
> So we are not "making this SR aware" so much as enabling "SR-over-foo" using
> SIDs to identify the path segments that are tunnels.
>
> The various ASes that provide connectivity between the Ingress and Egress
> Domains could each be constructed differently and use different technologies
> such as IP, MPLS with global table routing native BGP to the edge, MPLS IP
> VPN, SR-MPLS IP VPN, or SRv6 IP VPN. That is, the Ingress and Egress SR
> Domains can be connected by tunnels across a variety of technologies. This
> document describes how SR identifiers (SIDs) are use to identify the paths
> between the Ingress and Egress and the techniques in this document apply to
> routes of all AFI/SAFIs.

>From §5: "To achieve this, each Tunnel TLV in the Tunnel Encapsulation
attribute contains a Prefix SID sub-TLV [I-D.ietf-idr-tunnel-encaps]
for X."  But rfc9012 restricts the use of the Prefix-SID sub-TLV:

   [RFC8669] only defines behavior when the BGP Prefix-SID attribute is
   attached to routes of type IPv4/IPv6 Labeled Unicast
   [RFC4760][RFC8277], and it only defines values of the BGP Prefix-SID
   attribute for those cases.  Therefore, similar limitations exist for
   the Prefix-SID sub-TLV: it SHOULD only be included in a BGP UPDATE
   message for one of the address families for which [RFC8669] has a
   defined behavior, namely BGP IPv4/IPv6 Labeled Unicast [RFC4760]
   [RFC8277].  If included in a BGP UPDATE for any other address family,
   it MUST be ignored.

IOW, even though the overall mechanism could not be SR-specific, the
SR solution can't be implemented in a general way (without more
consideration).

Alvaro.

---

From: John Scudder <[email-masked]>
Date: Mon, 17 May 2021 20:54:56 +0000

Hi Adrian,

Comments in line below.

> [External Email. Be cautious of content]
> 
> 
> Hi John,
> 
> I don't know how to make this clearer except maybe using some red paint.

That would be exclusionary to the colo(u)r-blind.

> We would write...

If you want, you could expand the paragraph as you’ve suggested, but I don’t think it’s necessary — now that you’ve pointed out the paragraph, it’s clear enough. However, I think the document is still misleading and even inconsistent about this. Let me quote some other paragraphs to you.

Section 1:

   Segment Routing (SR) [RFC8402] is a protocol mechanism that can be
   used within a DC, and also for steering traffic that flows between
   two DC sites.  

The “steering traffic that flows between two DC sites” can easily be read as meaning, steering it *through* the backbone network. I take it your intent is to mean, steering it *over* the backbone network. 

                       In order for a source (ingress) DC that uses SR to
   load balance the flows it sends to a destination (egress) DC, it
   needs to know the complete set of entry nodes (i.e., GWs) for that
   egress DC from the backbone network connecting the two DCs.  Note
   that it is assumed that the connected set of DCs and the backbone
   network connecting them are part of the same SR BGP Link State (LS)
   instance ([RFC7752] and [I-D.ietf-idr-bgpls-segment-routing-epe]) so
   that traffic engineering using SR may be used for these flows.

The requirement that the sites *and the backbone network connecting them* must all be part of the same BGP-LS instance caused me to raise my eyebrows up into my hairline, but there it is in the text. This surprising assumption (most service providers do not, to my knowledge, allow their customers to consume their LSDB), plus “traffic engineering using SR may be used for these flows”, plus the sentence noted above, led me a long way down the garden path of thinking you were proposing end-to-end SR forwarding.

And then we have Section 4:

   When a remote GW receives a route to a prefix X it uses the Tunnel
   Egress Endpoint Sub-TLVs in the containing Tunnel Encapsulation
   attribute to identify the GWs through which X can be reached.  It
   uses this information to compute SR Traffic Engineering (SR TE) paths
   *across the backbone network*

(emphasis added). This serves to confirm my misapprehension that this is an exclusively SR solution. 

So now on the one hand, I accept that you were completely serious about the paragraph you quoted, and that I mentally elided, having been dazzled by the parts I just quoted. On the other hand, I wonder what I’m misunderstanding about the parts I’ve just quoted, or if I’m not misunderstanding them, how we can square this circle.

> What does "by default" mean in that context?

I wish I knew.

> 1. Forwarding plane trust model. Can packets get into the SR system? The answer to that remains, "No, because traffic MUST be filtered at the domain boundaries." This requires that the domain boundary is the interface between an SR-capable node, and a non-SR node. In this document all GWs and ASBRs are part of the SR domain connected by tunnels across the transit ASes (although the nodes in the transit ASes are not part of that domain).

I’m relieved to hear it; this relates to my earlier comments.

> I dare say that draft-farrel-spring-sr-domain-interconnect explains this better through examples, but the chairs of SPRING told us that that draft had no chance of progressing.
> 

I suspect if we can get me out of the swamp described in my responses to #1 above, this one will go away.

>> its definition in RFC 8402. Here’s that definition, from §2:

[blah, blah, blah]

> The confusion appears to arise purely from the terms "ingress SR domain" and "egress SR domain" which is our bad choice of words. "Site" would be a better word and I will scrub the document to use…

Works for me.

>> COMMENT:

[…]

> Indeed, just one (with potentially multiple tunnel encaps). We can make that explicit unless you can find a reason why advertising more than one would be beneficial.

I don’t see a use for it off the top of my head, although I suppose if you squinted hard enough at a dual-stack network you might find one. On the other hand, I haven’t tried to think through what the implications would be of two or more auto-discovery routes existing at the same time, e.g. what if they advertise different tunnel encaps?

> "describes the whole path including those parts that are within the Egress Site"

Great, I assume you’ll make that substitution.

> Ouch! Yes, that slipped through. It's SIDs all the way down.

OK. I guess some work is needed on "places each in an MPLS label stack sub-TLV” as well, then.

[…]

> I don't think we intended (or said) "BGP protocol mechanisms.”

Considering that all (or almost all) the exposures you detailed relate to BGP, I would think it implicit, although when I wrote “BGP protocol mechanisms” I was assuming the entire BGP ecosystem, i.e. BGP and its transport.

> But other protocol mechanisms *do* exist to protect BGP exchanges between peers. TLS or TCP-AO spring to mind.

TCP-AO doesn’t do anything towards confidentiality. TLS does but it’s not currently used as a BGP transport, to my knowledge. Of course one can also run BGP over IPSec if one wants confidentiality, and use of other transports could be proposed. But other than running it over IPSec, I’m not aware of “existing protocol mechanisms” to avoid “exposures of information” relating to BGP.

You might well say “ok fine, use IPSec then”. But that’s not a complete answer either. For example, there are at least two different parties operating different parts of the network — the site owner, and the backbone owner. It’s likely challenging for one to mandate the other’s security practices. 

> The point here, is that it is not the job of this document to solve BGP security issues.

I agree. I just think you’re overpromising by saying "SHOULD be protected against using existing protocol mechanisms”. If you removed those words, I don’t think I’d have a problem. If you want to offer some hope and not leave the reader depressed you could even finesse it by saying something other than “existing protocol mechanisms”, e.g. if you mean the operator(s) should use a transport that provides confidentiality, say so. If you think it’s a sad mad bad world we live in and that BGP’s security model needs some work, say so, or remain silent, but don’t imply that if the operator just turned on the right knob they’d be fine.

> Weeeeeeell, I think that that IDR work will need to provide adequate security.

Fair enough, I’ll lob that over the fence to them.

> That is really what we intended you to wonder about. I suppose we are saying, "We are worried it might have an effect, but we can't put our finger on it."

OK I guess.

> Further, if GW1 can no longer receive advertisements from GW2 then it will stop advertising on behalf of GW2.

Replied separately.

Thanks,

—John

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 17 May 2021 17:04:44 -0400

Hi John

I agree with your comments that the scenario I mentioned is covered in
Section 3 and agree as well on the RFC 2119 keyword usage scrub.

In-line

On Mon, May 17, 2021 at 3:55 PM John Scudder <[email-masked]> wrote:

> > So if GW2 connection to external was down but GW1 still has its
> connection to external.  GW2 would auto discover GW1 over iBGP and GW2
> would advertise both GW1 and GW2 as reachable gateways.  However GW2 has
> its external peer down.  So if GW1 continues to advertised GW2 as we stated
> GW1 will auto discover  GW2 over iBGP.
>

   Gyan> Yes.  Agreed.  I wanted to draw some more attention to this to the
authors on the withdrawal that it’s critical and agreed a MUST.

>
>
> So when GW2’s external peering goes down, GW2 withdraws its auto discovery
> route, and therefore GW1 re-advertises its routes externally without GW2
> listed in the tunnel attribute.
>

> I will say that reviewing the above-quoted text — which seems tailor-made
> for a “MUST withdraw” — made me notice that the draft makes only sporadic
> and desultory use of RFC2119 keywords. In fact there are so few used, that
> it seems like it might be better to scrub those two SHOULD and two MUST out
> and remove the 2119 citation.

Gyan> Agreed.  I will parse the draft for RFC 2119 keyword  placement  in
my final GEN-ART review update

-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 17 May 2021 18:48:57 -0400

Adrian

In the introduction it mentions the following backbone transport:

  In the draft it talks about SR steering

      however can RSVP be used in the backbone transport .

I think to clarify we should mention the MUST for the data plane
requirements for the backbone transport.

On the call last week we confirmed  that the ingress and egress
domains now called “sites” not domains do not have to be SR enabled.

Kind Regards

Gyan

On Mon, May 17, 2021 at 5:04 PM Gyan Mishra <[email-masked]> wrote:

>    Gyan> Yes.  Agreed.  I wanted to draw some more attention to this to
> the authors on the withdrawal that it’s critical and agreed a MUST.
>
>>
>>
>> So when GW2’s external peering goes down, GW2 withdraws its auto
>> discovery route, and therefore GW1 re-advertises its routes externally
>> without GW2 listed in the tunnel attribute.
>>
>
>
>
>

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 17 May 2021 20:23:34 -0400

Alvaro

Very good points brought up on the limitations on the tunnel encapsulation
attribute BGP prefix sid sub tlv.  Can only be used with BGP LU AFI / SAFI.

Kind Regards

Gyan

On Mon, May 17, 2021 at 4:12 PM Alvaro Retana <[email-masked]>
wrote:

> technologies
> We
> > would write...
> technologies

Gyan> RFC8669 limitation

3.1 <https://datatracker.ietf.org/doc/html/rfc8669#section-3.1>.
Label-Index TLV

   The Label-Index TLV MUST be present in the BGP Prefix-SID attribute
   attached to IPv4/IPv6 Labeled Unicast prefixes ([RFC8277
<https://datatracker.ietf.org/doc/html/rfc8277>]).  It MUST
   be ignored when received for other BGP AFI/SAFI combinations.

RFC 9012 Tunnel encapsulation attribute

3.7 <https://datatracker.ietf.org/doc/html/rfc9012#section-3.7>.
Prefix-SID Sub-TLV (Type Code 11)

   [RFC8669] defines a BGP path attribute known as the "BGP Prefix-SID
   attribute".  This attribute is defined to contain a sequence of one
   or more TLVs, where each TLV is either a Label-Index TLV or an
   Originator SRGB (Source Routing Global Block) TLV.

   This document defines a Prefix-SID (Prefix Segment Identifier) sub-
   TLV.  The Value field of the Prefix-SID sub-TLV can be set to any
   permitted value of the Value field of a BGP Prefix-SID attribute
   [RFC8669 <https://datatracker.ietf.org/doc/html/rfc8669>].

   [RFC8669] only defines behavior when the BGP Prefix-SID attribute is
   attached to routes of type IPv4/IPv6 Labeled Unicast
   [RFC4760 <https://datatracker.ietf.org/doc/html/rfc4760>][RFC8277],
and it only defines values of the BGP Prefix-SID
   attribute for those cases.  Therefore, similar limitations exist for
   the Prefix-SID sub-TLV: it SHOULD only be included in a BGP UPDATE
   message for one of the address families for which [RFC8669
<https://datatracker.ietf.org/doc/html/rfc8669>] has a
   defined behavior, namely BGP IPv4/IPv6 Labeled Unicast [RFC4760
<https://datatracker.ietf.org/doc/html/rfc4760>]
   [RFC8277 <https://datatracker.ietf.org/doc/html/rfc8277>].  If
included in a BGP UPDATE for any other address family,
   it MUST be ignored.

-- 

*Gyan Mishra*

*M 301 502-1347*