Subject: Re: [bess] Benjamin Kaduk's Discuss on draft-ietf-bess-datacenter-gateway-11: (with DISCUSS and COMMENT)

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 21 Jul 2021 10:05:47 -0700

Oops, this got set aside for much longer than planned (even with the
multiple reminders from the authors).
My humble apologies for the long delay and the need to be reminded.
In part it seems that I misremembered how complicated my concerns were and
thus was trying to allocate a larger chunk of time to reviewing the changes
than ended up being necessary, but that only accounts for a small part of
the delay.

Carrying on from where we left off, which is I suppose the best I can do at
this point...

On Wed, Jun 02, 2021 at 12:00:57PM +0100, Adrian Farrel wrote:
> Thanks for what you got to yesterday.
> 
> We can take this step by step.
> 
> [snip the Discuss]
> 
> >>    <t>Given that the gateways and ASBRs are connected by tunnels that may run across parts of the network that are not trusted, 
> >>         data center operators using the approach set out in this network SHOULD consider using gateway-to-gateway encryption to
> >
> > "SHOULD consider"?  We're not even bold enough to go with just SHOULD?
> 
> I find this aspect of IETF Security considerations hard. Maybe I have a philosophical problem.
> You might tell your child or another vulnerable person that they "SHOULD carry their money in a safe place". But with an adult one usually points out the risks and says that they "SHOULD consider carrying their money in a safe place."
> 
> >>    protect the data center traffic.  Additionally, due consideration SHOULD be given to encrypting end-to-end traffic as it
> >>    would be for any traffic that uses a public or untrusted network for transport.</t>
> >
> > I am thinking that I might be in a similar situation as Warren: my
> > recollection from when we processed 8402 is that nodes remotely connected
> > to each other would be using secure tunnels that provide at least as much
> > protection as the "secure physical network" within the local domain. 
> 
> I really don't want to be painted as someone who supports every word of 8402, but...
> 
> The essence of 8402 is that intervening transit nodes do not need to be tunnelled past.
> 
> Consider, for example, how SRv6 packets may be forwarded across regular IPv6 (i.e., not SRv6-capable) routers. The sending SRv6 router sets the IPv6 destination address to the address of the receiving SRv6 router, updates the SRH (which is just an IPv6 extension header), and forwards the packet. The transit routers simply forward the packet as an IPv6 packet without even being aware that there is an SRH present. Only when the packet arrives at the receiving SRv6 router (which sees itself in the IPv6 destination field) is SRv6 actioned. Thus, there may be a "logical tunnel" between remotely connected SRv6 nodes, but there is no tunnelling technology used. Thus, the protection is weaker than in a secure physical network.
> 
> Now, you and I might argue that this gives a strong incentive to use a secure tunnelling technology between remotely connected SR nodes. That is, however, hard for the sender to enforce for remote (sic) remotely connected nodes. Hence the consideration of GW-to-GW encryption. 
> 
> The proponents of 8402 might argue that SR is a forwarding technology and so it is no different to IP: it is up to the applications to decide whether the underlying network is safe. Thus, if a service provider decides to use SR inside their network, it has no additional security considerations over the same provider using IP inside their network.

That's fair, and thank you for writing it out.
(You and I might also argue that there is strong incentive to use a secure
tunnelling technology between remotely connected nodes even when non-SR IP
is in use...)

> > With
> > that background, I would have gone with MUSTs here, but I am not at the
> > moment finding much in 8402 to strongly support that as a requirement.
> 
> Well, I could go with "MUST consider" and "consideration MUST be given". That directs thought, but not action.

Right.

And as this is the end of the DISCUSS section, I'll note that the text in
the -12 does resolve my listed concerns; thank you!

> >>> COMMENT:
> >>> ----------------------------------------------------------------
> >>>
> >>> The Abstract is perhaps pushing the bounds of reasonable length for an abstract. 
> >> 
> >> The Style Guide recommends 25 lines or fewer. We have 18 lines of text.
> >
> > Er, which style guide?  I'm not seeing much in RFC 7322 ... and
> > https://www.rfc-editor.org/policy.html#policy.abstract (which admittedly
> > does say it has been replaced by https://www.rfc-editor.org/styleguide/)
> > puts the cap at 20, with 5-10 being "typical".
> 
> Oh, I apologise. 
> Either my memory is faulty or this section has been updated over the last twenty years.
> Anyway, "cap at 20" tells us what to do.
> 
> [snip]
> 
> >> NEW
> >>    Data centers are attached to the Internet or a backbone network by
> >>    gateway routers.  One data center typically has more than one gateway
> >>    for commercial, load balancing, and resiliency reasons.  Other sites,
> >>    such as access networks, also need to be connected across backbone
> >>    networks through gateways.
> >> 
> >>    This document defines a mechanism using the BGP Tunnel Encapsulation
> >>    attribute to allow data center gateway routers to advertise routes to the 
> >>    prefixes reachable in the site, including advertising them on behalf of 
> >>    other gateways at the same site.  This allows segment routing to be used
> >>    to identify multiple paths across the Internet or backbone network 
> >>    between different gateways.  The paths can be selected for load-balancing,
> >>    resilience, and quality purposes.   
> >> END
> >
> > Thanks, this is an improvement over the OLD version.
> 
> And only 12 lines 😊
> 
> >>> Section 1
> >>>
> >>>   The solution described in this document is agnostic as to whether the
> >>>   transit ASes do or do not have SR capabilities.  the solution uses SR
> >>>   to stitch together path segments between GWs and through the ASBRs.
> >>>   Thus, there is a requirement that the GWs and ASBRs are SR-capable.
> >>>   The solution supports the SR path being extended into the ingress and
> >>>   egress sites if they are SR-capable.
> >>>
> >>> There seem to be some nodes marked "ASBR" that are at the boundary 
> >>> between the two transit ASes, in Figure 1.  This text leaves me uncertain
> >>> whether they are expected to support SR (vs just the ASBRs that are 
> >>> attachment points for the ingress/egress GWs).
> >> 
> >> Whether the interior nodes of the transit ASes support SRs doesn't really
> >> matter to us. The GWs and ASBRs are interconnected by tunnels as
> >> represented by links in the BGP-LS instance. SR is used to stitch these 
> >> tunnels together to create an end-to-end path between an ingress site
> >> and an egress site. Thus, the GWs and ASBRs must be SR capable in order
> >> to participate.
> >
> > I think we should say something about how the ASBRs that are "abstracted
> > away" by the tunnel are not relevant, since otherwise just saying "ASBR"
> > would seem to include them.  (They are listed in the figure as ASBRs, after
> > all.)
> 
> The tunnels are not GW to GW, they are ASBR to ASBR. So there is a concatenation of tunnels between the GWs.
> 
> It is possible that a tunnel might span several ASes, but that would be unusual: at the network layer ASes tend to terminate and restart tunnels.
> 
> I don't see anything in the document that talks about "abstracting away" ASBRs. If there is something I've missed then we should fix it.
> 
> You may find draft-spring-farrel-sr-domain-interconnect helpful to see how the SR SID list can be constructed to make all this work.
> 
> The challenge here, I think, is that this is supposed to be a short document that describes the BGP extensions, not a whole cookbook.

I think I have finally managed to get what is going on here through my
thick skull.  Thank you for your patience while I did so; I now think the
current text is fine as-is.

> >>> Section 3
> >>>
> >>>   o  Each GW is configured with an identifier for the site.  That
> >>>      identifier MUST be the same across all GWs to the site (i.e., the
> >>>      same identifier is used by all GWs to the same site), and MUST be
> >>>      unique across all sites that are connected (i.e., across all GWs
> >>>      to all sites that are interconnected).
> >>>
> >>> The advice in draft-gont-numeric-ids-sec-considerations is probably 
> >>> relevant here.  How should we pick these identifiers?  Which properties
> >>> are necessary and which are not needed? 
> >> 
> >> This bullet needs to be taken along with the bullet that follows, viz.
> >> 
> >>    o  A route target ([RFC4360]) MUST be attached to each GW's auto-
> >>       discovery route (defined below) and its value MUST be set to the
> >>       site identifier.
> >> 
> >> We will re-jig these two bullets to say...
> >> 
> >>    o  A route target ([RFC4360]) MUST be attached to each GW's auto-
> >>       discovery route (defined below) and its value MUST be set to a 
> >>       value that identifies the site identifier.  The rules for constructing
> >
> > (nit) "identifies the site identifier" is probably one too many "identifie"s
> 
> Ack. "indicated the site identifier"
> 
> >>       a route target are detailed in [RFC4360].  It is RECOMMENDED that
> >>       a Type x00 or x02 route target be used.
> >> 
> >>    o Site identifiers are set through configuration.  The site identifiers 
> >>       MUST be the same across all GWs to the site (i.e., the same
> >>       identifier is used by all GWs to the same site), and MUST be
> >>       unique across all sites that are connected (i.e., across all GWs
> >>       to all sites that are interconnected).
> >
> > This helps, but I still wonder if we can give some guidance to the operator
> > on how to choose the value of the (configured) site identifier.  If
> > everyone decides to label their sites as consecutive integers starting at
> > one, for example, that is likely to end in sadness later on.
> 
> You are asking us to give guidance on how to set route target values.
> 4360 says...
>    When the value of the high-order octet of the Type field is 0x00 or
>    0x02, the Local Administrator sub-field contains a number from a
>    numbering space that is administered by the organization to which the
>    Autonomous System number carried in the Global Administrator sub-
>    field has been assigned by an appropriate authority.
> 
>    When the value of the high-order octet of the Type field is 0x01, the
>    Local Administrator sub-field contains a number from a numbering
>    space that is administered by the organization to which the IP
>    address carried in the Global Administrator sub-field has been
>    assigned by an appropriate authority.
> ...This is carefully set up so that clashes of RT value can only arise from misconfiguration within an administration.
> We are recommending using type 0 or 2, so that the RT is scoped by AS and easily administered by the AS operator. 

What's in the -12 (which I think is basically what's quoted here) works
well.

The below stuff all sounds good, too; thanks for all the responses and
updates.

Now to go finish up going through the rest of your initial reply, which
should have been done a month or two ago :(

-Ben

> >>> o  Each GW MUST construct an import filtering rule to import any
> >>>      route that carries a route target with the same site identifier
> >>>      that the GW itself uses.  This means that only these GWs will
> >>>      import those routes, and that all GWs to the same site will import
> >>>      each other's routes and will learn (auto-discover) the current set
> >>>      of active GWs for the site.
> >>>
> >>> This seems pretty fragile in the face of identifier collisions; I hope there
> >>> is some good text in the security considerations that covers the risks here.
> >>> [ed. it seems we cover other aspects relating to identifier selection but not
> >>> this one] Is there any filtering that can be done other than by site identifier,
> >>> e.g., to know that a certain peer would never be able to advertise something
> >>> that validly has the same site identifier?
> >> 
> >> This mechanism to control the importing of routes (using route targets) has
> >> been in use in BGP networks for a while (more than 15 years since RFC 4360).
> >> 
> >> One of the big uses for route targets is in identifying which sites belong to the
> >> same VPN (see RFC 4364). There is the potential to misconfigure the route
> >> target used to identify a set of VPN sites by giving one site the wrong value.
> >> That would mean that it:
> >> - fails to import the relevant routes
> >> - doesn't get its routes imported by its peers
> >> - may accidentally import false routes
> >> - may accidentally have its routes imported by other sites in different VPNs
> >> That is all "bad stuff" and 4364 contains no mechanisms to handle what happens
> >> if there is a misconfiguration (it's not an attack vector unless BGP or a router is
> >> compromised - in which case far worse stuff happens). Basically, if you mess up
> >> the configuration, expect the unexpected.
> >
> > Agreed, this is all "bad stuff", and the latter two are more what I had in
> > mind when I wrote the comment.  It's part of my job to wonder about the
> > potential scope of fallout if BGP or a router is compromised, even if we
> > expect that to not actually happen much in practice :)
> 
> I don't make any claims about whether a BGP router might be compromised. But I do say that if it is, then some really bad stuff will happen, not just some site traffic being misdelivered.
> Perhaps we should spend more time worrying about compromised routers - it's not a new discussion! 
> 
> We seem to have survived the possibility of VPN misconfiguration for a number of years. There may be some work that could be done to "autodetect" such misconfiguration.
> 
> [snip]
> 
> >>>   As described in Section 1, each GW will include a Tunnel
> >>>   Encapsulation attribute with the GW encapsulation information for
> >>>   each of the site's active GWs (including itself) in every route
> >>>   advertised externally to that site.  [...]
> >>>
> >>> (I assume this is not intended to preclude the usual route filtering/split-
> >>> horizon type stuff.)
> >> 
> >> Definitely doesn't preclude it. In fact that rule remains key to the way 
> >> GWs behave - they don't re-advertise routes they have learned from
> >> other GWs. But (of course) all the GWs to a site advertise the same
> >> routes. 
> >
> > I had to read this twice just to make sure I'm picking up on the details:
> > the GWs of a site do *not* advertise the *routes* they learn from each
> > other; the only re-advertising is of GW encapsulation information.
> > I assume that John has covered the case of intra-site connectivity issues
> > whereby this could entail a site advertising a route with encapsulation
> > information that couldn't actually get to that prefix, and will not think
> > about it further.
> 
> Yes, that was part of the discussion with John.
> 
> Thanks,
> Adrian
>

---

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 21 Jul 2021 10:18:48 -0700

Picking up (belatedly) where I left off in my initial reply...

On Fri, May 28, 2021 at 10:03:12PM +0100, Adrian Farrel wrote:
[snip]
> >                                        As the current set of active GWs
> >   changes (due to the addition of a new GW or the failure/removal of an
> >   existing GW) each externally advertised route will be re-advertised
> >   with a new Tunnel Encapsulation attribute which reflects current set
> >   of active GWs.
> >
> > The "everybody advertises the union of what they've seen" behavior seems
> > like it will latch NLRI in place as being a GW, but here we're saying that
> > removal will be propagated as well as addition.  What's the mechanism for
> > removing stale data (whether maliciously added or as part of maintenance? 
> > If it's an explicit withdrawal, is that also propagated by everybody?  How long
> > does it have to stay around for?  (I recognize that some of this is just stock
> > BGP, but I am looking for more clarity on how it interacts with the "advertise
> > the union of what you saw" behavior that is new to this document.
> 
> Yes, this is all handled by standard BGP mechanisms. It's how the withdraw message works and is propagated.
> 
> If a gateway auto-discovery route gets withdrawn (explicit message or dropped peering), then the remaining gateways remove its tunnel TLVs from the union and re-advertise the site's routes. 

For what it's worth, what's going on here seems to have become much more
clear to me after a re-read of the document and the month's delay.  I'm
sorry that I was confused about it the first time around.

In short: the "union of what you saw" only gets advertised externally, and
the auto-discovery (internal) advertisements will always just be for the
advertising router's own information.  There's no auto-discovery that
includes the union of what you saw, which is I think what I was concerned
about here.

> > The text in the next paragraph mentions that there can be situations with
> > broken internal routing where things land in a broken state -- how long do
> > they stay broken and how can they be fixed?
> 
> It completely depends upon the situation, what the breakage is, and how the breakage is discovered. 
> JGS suggested we flag up the situation rather than try to solve it (which we did in the paragraph quoted below).
> While not an impossible situation, it does represent a strange brokenness that is possibly causing traffic within the site to get misrouted as well.
> 
> >   If a gateway becomes disconnected from the backbone network, or if
> >   the site operator decides to terminate the gateway's activity, it
> >   MUST withdraw the advertisements described above.  This means that
> >   remote gateways at other sites will stop seeing advertisements from
> >   this gateway.  Note that if the routing within a site is broken (for
> >   example, such that there is a route from one GW to another, but not
> >   in the reverse direction), then it is possible that incoming traffic
> >   will be routed to the wrong GW to reach the destination prefix - in
> >   this degraded network situation, traffic may be dropped.
> >
> > This is probably worth reiterating in the security considerations section.
> 
> Muttering about, "Not all problems are security problems" 😊
> As an attack it says, "If you can break the routing within a site, then traffic coming from outside the site might also be incorrectly routed."
> We don't think reiteration of this peculiarly broken situation would help.

Ok.

> >   Note that if a GW is (mis)configured with a different site identifier
> >   from the other GWs to the same site then it will not be auto-
> >   discovered by the other GWs (and will not auto-discover the other
> >   GWs).  This would result in a GW for another site receiving only the
> >   Tunnel Encapsulation attribute included in the BGP best route; i.e.,
> >   the Tunnel Encapsulation attribute of the (mis)configured GW or that
> >   of the other GWs.
> >
> > Are there noteworthy operational considerations of this, e.g., if all the
> > traffic gets directed to a GW that lacks the bandwidth to handle it?
> 
> It is worth noting that without this mechanism, all traffic gets directed to just one gateway because that is what BGP does (plus or minus ADDPATH). 
> What this document does is allow path selection, and choosing paths allows a degree of traffic engineering. So the misconfiguration situation is never worse than before this document.

Good point!

> > Section 4
> >
> >   attribute to identify the GWs through which X can be reached.  It
> >   uses this information to compute SR Traffic Engineering (SR TE) paths
> >   across the backbone network looking at the information advertised to
> >   it in SR BGP Link State (BGP-LS)
> >
> > This seems to leave the reader wondering about the details of how 
> > those SR TE paths are computed.  I understand that it's properly out
> > of scope for this document, but a reference would go a long way.
> 
> We appreciate the inquisitive reader! In the Introduction we say...
> 
>    The solution defined in this document can be seen in the broader 
>    context of site interconnection in [I-D.farrel-spring-sr-domain-interconnect].
>    That document shows how other existing protocol elements may be
>    combined with the solution defined in this document to provide a full
>    system, but is not a necessary reference for understanding this document.
> 
> > Section 5
> >
> >   for a prefix X, then each GW computes an SR TE path through that site
> >   to X from each of the currently active GWs, and places each in an
> >   MPLS label stack sub-TLV [RFC9012] in the SR Tunnel TLV for that GW.
> >
> > I don't think I understand why each (egress) GW has to (re)compute
> > the path through the site to X for each of the GWs at the site -- can't
> > it just take the sub-TLV it got from the peer and re-propagate it?
> 
> Oh, it doesn't have to recompute it *if* it is present (i.e. if it got it from the peer). But that part of the path might not be present (that is, the sub-TLV might not be present) because:
>     a. the ingress might not have visibility into the egress site beyond simple reachability 
>     b. the ingress might not care 
>     c. the ingress might want to let the egress site make subtle reactive choices according to local conditions
> IMHO it would be unusual for the tail end of the path to be specified in this way.

(Disclaimer: this is not an important topic and I'm happy to drop it for
expediency if desired.)  I am not sure that my point came across as
intended.  The text here seems to be about what the egress GW does when it
advertises the "union of all tunnel encapsulation information" route
externally.  My understanding/expectation was that each egress GW would be
able to just take the bits it got from auto-discovery (internal) routes and
squish them together.  As you note, the ingress may not care or need the
details of the path within the egress site from GW to prefix X, and so I
don't understand why the advertising egress GW would go to the trouble of
computing a route from the other GWs in its site to prefix X.  If such a
route was needed, the other GW in the site would be better placed to
compute such a route and include it in the auto-discovery route anyway.

> BTW, the computation is not done per packet. Just like regular routing inside the site, it is done according various state-based triggers, and stored for the destination prefix.

(Right.)

> > Section 6
> >
> > [The topic of which sites are allowed to send in the site's native encapsulation seems
> > related to questions of what an "SR Domain" is and what boundary security it has.  I
> > think that the other ADs are basically covering this topic, though, so am not sure there
> > is much more to say here.]
> >
> >   If the GWs for a given site are configured to allow remote GWs to
> >   send them a packet in that site's native encapsulation, then each GW
> >   will also include multiple instances of a Tunnel TLV for that native
> >   encapsulation in externally advertised routes: one for each GW and
> >   each containing a Tunnel Egress Endpoint sub-TLV with that GW's
> >   address.  [...]
> >
> > Does this implicitly require that all the GWs of the site have the same configuration
> > for whether or not to allow native encapsulation from remote GWs?  How would
> > things degrade if a mixed configuration did happen to occur?
> 
> This applies GW by GW since the tunnels lead to a GW, and the path has selected the tunnels. So, the sender knows.
> 
> If a gateway is configured to not allow native encapsulation, then it will receive packets in some other encapsulation that it does understand, and it will convert them to the site's native encapsulation. 
> 
> Note that the GW is part of the site, so it really (really) needs to understand the native encapsulation in the site!
> 
> Note also that (of course?) a GW that receives packets in an encapsulation it doesn't understand (and hasn't advertised that it understands) will drop the packets (just like any other data plane node will discard packets with an unknown encapsulation).
> 
> Well, there is a case where a GW advertises that it understands an encapsulation, but actually doesn't understand it. That is at best a bug, and at worst a purchasing error.

Okay.  So if there is an issue here at all (not entirely clear) it's just
an editorial one about "the GWs for a given site" vs "any GWs for a given
site", and "each GW" vs "each such GW" (or similar).

> > Section 8
> >
> >   From a protocol point of view, the mechanisms described in this
> >   document can leverage the security mechanisms already defined for
> >   BGP.  Further discussion of security considerations for BGP may be
> >   found in the BGP specification itself [RFC4271] and in the security
> >   analysis for BGP [RFC4272].  The original discussion of the use of
> >   the TCP MD5 signature option to protect BGP sessions is found in
> >   [RFC5925], while [RFC6952] includes an analysis of BGP keying and
> >   authentication issues.
> >
> > Such an elegant way of not mentioning TCP-AO :) (I do see that it is actually referenced, just not mentioned by name.)
> 
> Hmmm. The document that contains TCP-AO is referenced, but that is because of the useful discussion of MD5 that it contains.
> 
> > The whole section is quite nicely done, actually -- thank you!
> 
> We aim to please.
> 
> > Section 11
> >
> > I don't really understand why draft-ietf-idr-bgpls-segment-routing-epe
> > is listed as normative but draft-ietf-idr-bgp-ls-segment-routing-ext is 
> > listed as informative.  They seem to be used in the same place.
> 
> Oops, the former draft should be listed as Informative

It is good to know that not all of my confusion is misguided :)

> > NITS/EDITORIAL

No further comments on the nits/editorial, but the extra commentary is
appreciated.

I will try to remember to get you the chocolate-sprinkles option if I'm
ever faced with that choice...

Thanks again for all your care in the responses and updates, as well as
your patience with my laggardly replies.  Any strain in such patience is
justified, and I accept the blame for it.

I will go post a No Objection ballot on the -12 now.

-Ben

> > Section 1
> >
> >   two DC sites.  In order for a source DC (also known as an ingress DC)
> >   that uses SR to load balance the flows it sends to a destination DC
> >
> > I'd consider "also known as an ingress DC since it forms the ingress endpoint of a tunnel".
> 
> Pedant alert - the site doesn't form the ingress of the tunnel, that is the gateway.
> But thanks for making me re-read because this should be "site" not "DC"!
> 
> >   sites could each be constructed differently and use different
> >   technologies such as IP, MPLS with global table routing native BGP to
> >   the edge, MPLS IP VPN, SR-MPLS IP VPN, or SRv6 IP VPN.  That is, the
> >
> > FWIW I don't think I figured out what "MPLS with global table routing
> > native BGP to the edge" means with any real confidence, and my attempts
> > to google it basically just found this document.  So please feel encouraged
> > to take another look at the phrasing.  My current most-likely interpretation
> > is that there is internally MPLS with global table, but that what's presented 
> > to the outside is native BGP-based IP routing, so there's some implied
> > translation layer.
> 
> Yeah, "MPLS inside the AS based on externally visible address reachability like what IP does." That's basically how MPLS-enabled ASes work.
> 
> > Section 3
> >
> >   To avoid the side effect of applying the Tunnel Encapsulation
> >   attribute to any packet that is addressed to the GW itself, the GW
> >   MUST use a different loopback address for packets intended for it.
> >
> > "different" is most clear when we list both things that differ.
> > So perhaps it's safer to say that the address advertised for auto-
> > discovery must use a different loopback address than is advertised
> > for packets directed to the gateway itself.
> 
> OK
> 
> > Section 5
> >
> >   achieve this, each Tunnel TLV in the Tunnel Encapsulation attribute
> >   contains a Prefix SID sub-TLV [RFC9012] for X.  As defined in
> >   [RFC9012], the Prefix SID sub-TLV is only for IPv4/IPV6 labelled
> >
> > I wonder if this sentence break should really be a paragraph break, 
> > since the following paragraph seems to cover MPLS in a way that
> > roughly parallels how we treat IP here.
> 
> Ack
> 
> >   applies to routes of those types.  If the use of the Prefix SID sub-
> >   tlv for routes of other types is defined in the future, further
> >   documents will be needed to describe their use.
> >
> > I think that we are missing a "for SR TE tunnel encapsulation" at the end,
> > as the current text is basically saying "if the use of X is defined in the future,
> > future documents will be needed to describe the use of X", which is fairly
> > devoid of content.
> 
> Ah, no.
> Other Prefix-SID sub-TLVs might be defined for general use in SR.
> If that happens, further documents will also be needed to describe their use in the context of this document.
> Rephrased.
> 
> >   Alternatively, if MPLS SR is in use and if the GWs for a given site
> >   are configured to allow remote GWs to perform SR TE through that site
> >   for a prefix X, then each GW computes an SR TE path through that site
> >
> > We might benefit from sprinkling around some "ingress" and "egress"
> > here.
> 
> Prefer chocolate sprinkles, but have reluctantly used ingress and egress.
>

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 22 Jul 2021 19:12:50 +0100

> Picking up (belatedly) where I left off in my initial reply...

Thanks, Ben.

>> Oh, it doesn't have to recompute it *if* it is present (i.e. if it got it from
>> the peer). But that part of the path might not be present (that is, the 
>> sub-TLV might not be present) because:

I think you may have misunderstood Section 5. 
Firstly, the case you are looking at is behind a cascaded "if".
But also, it doesn't say that each GW computes paths from each other gateway to the destination. It only says that each GW computes a path from itself to the destination. Each GW then encodes that path as MPLS SID stack, puts it in an MPLS label stack sub-TLV inside the SR tunnel TLV. And, of course, that means that when a GW advertises the reachability on behalf of the other GWs to the site, it will pick up those routes and advertise them as well.

>> This applies GW by GW since the tunnels lead to a GW, and the path has
>> selected the tunnels. So, the sender knows.
>> 
>> If a gateway is configured to not allow native encapsulation, then it will
>> receive packets in some other encapsulation that it does understand,
>> and it will convert them to the site's native encapsulation. 
>> 
>> Note that the GW is part of the site, so it really (really) needs to
>> understand the native encapsulation in the site!
>> 
>> Note also that (of course?) a GW that receives packets in an
>> encapsulation it doesn't understand (and hasn't advertised that it
>> understands) will drop the packets (just like any other data plane
>> node will discard packets with an unknown encapsulation).
>> 
>> Well, there is a case where a GW advertises that it understands
>> an encapsulation, but actually doesn't understand it. That is at
>> best a bug, and at worst a purchasing error.
>

I'm pretty sure that the text is clear enough here. 
I suppose it is the site that is configured, and all gateways act the same way.
So we could have
   If a site is configured to allow remote GWs send packets to the site in
   the site's native encapsulation, then each GW to the site will also 
   include multiple instances of a Tunnel TLV for that native encapsulation
   in externally advertised routes: one for each GW and each containing a
   Tunnel Egress Endpoint sub-TLV with that GW's address.  [...]

I'll make that change if you think it is important.

Cheers,
Adrian

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 22 Jul 2021 11:22:35 -0700

> I think you may have misunderstood Section 5. 

It seems like I did, yes.  Sorry about that.

> But also, it doesn't say that each GW computes paths from each other gateway to the destination. It only says that each GW computes a path from itself to the destination. Each GW then encodes that…

(This, of course, matches up with the behavior I expected.)

> I'll make that change if you think it is important.

I don't think it's important to make the change, though I do think the
change is an improvement.

Thanks,

Ben