Subject: [GROW] draft-ietf-grow-bgp-gshut

From: <[email-masked]>
Date: Thu, 22 Jun 2017 10:03:59 +0000

Hi all,

 > From: Job Snijders [mailto:[email-masked]]
 > Sent: Friday, May 26, 2017 4:05 PM
> 
 > Hi GROW,
 > 
 > I've compiled a todo list to outline the next steps for the
 > draft-ietf-grow-bgp-session-culling document.
 > 

[...]
 
 > BGP g-shut (possible action for Bruno et al)
 > --------------------------------------------
 > 
 > Bruno promised a new, fresh version of draft-ietf-grow-bgp-gshut which
 > would focus on the rfc1997 well-known community 65535:0 - I would
 > appreciate if this is posted at some point so we can make an Informative
 > reference to a non-zombie draft.  NTT (as2914) & Coloclue (as8283)
 > implemented experimental rfc1997 gshut support for customers and it
 > appears to work as advertise (no pun intended ;-).
 
https://tools.ietf.org/html/draft-ietf-grow-bgp-gshut-07 has just been posted and addresses the above point.

As previously discussed on the list, the main technical change is the focus, for the eBGP signaling, on the use of a well-known BGP community.
In other word, the alternative option to use a non-transitive community has been removed, following latest IDR discussion on community and the lack of implementation (hence IDR WG progress) of draft-ietf-idr-reserved-extended-communities / draft.ietf-idr-as4octet-extcomm-generic-subtype.
As a benefit, this draft is now aligned with:
- the BGP gshut/planned maintenance implementations disclosed on this list https://mailarchive.ietf.org/arch/msg/grow/ReJtavkJDlyrDo5qoD7u9iJDLXs
- the deployed usage

Thanks to Job for the in-depth review, comments, and reminders to update the draft.

--Bruno

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Job Snijders <[email-masked]>
Date: Thu, 22 Jun 2017 22:47:05 +0200

Dear working group,

> As previously discussed on the list, the main technical change is the
> focus, for the eBGP signaling, on the use of a well-known BGP
> community.  In other word, the alternative option to use a
> non-transitive community has been removed, following latest IDR
> discussion on community and the lack of implementation (hence IDR WG
> progress) of draft-ietf-idr-reserved-extended-communities /
> draft.ietf-idr-as4octet-extcomm-generic-subtype.  As a benefit, this
> draft is now aligned with: - the BGP gshut/planned maintenance
> implementations disclosed on this list
> https://mailarchive.ietf.org/arch/msg/grow/ReJtavkJDlyrDo5qoD7u9iJDLXs
> - the deployed usage

fantastic!

I have some comments:

Summary:

I think that the neighbor ASBR should _not_ strip the GSHUT well-known
community, additionally, the place where the low local preference is set
should move closer to the initiator of the gshut.  Instead of setting
the low LP on Adj-RIB-Out to IBGP neighbors, the low LP should be set
during application of the local policy on the relevant Adj-RIB-In.

The above changes would align with current operational practises, i've
learned from numerous people that they use AS-specific "lower LOCAL_PREF
as low as possible" communities from their peers and upstream providers
to trigger path hunting. In current deployments those communities are
often not stripped (in context of IBGP), and the lower LOCAL_PREF is set
on "ebgp-in" rather then "ibgp-out".

Optionally an appendix with an actual router configuration can be
supplied, if there is interest in this I am happy to provide that.

-----
Suggestions:

OLD Abstract:
   This draft describes operational procedures aimed at reducing the
   amount of traffic lost during planned maintenances of routers or
   links, involving the shutdown of BGP peering sessions.
NEW Abstract:
   This draft describes operational procedures aimed at reducing the
   amount of traffic lost during planned maintenances of routers or
   links, involving the shutdown of BGP peering sessions. Additionally
   this document describes the use of a well-known Border Gateway
   Protocol (BGP) community to signal that a graceful shutdown has been
   initiated for the tagged prefix.

OLD (in introduction):
   Still, it explains why reserving a community value for the purpose of
   BGP session graceful shutdown would reduce the management overhead
   bound with the solution.  It would also allow vendors to provide an
   automatic graceful shutdown mechanism that does not require any
   router reconfiguration at maintenance time.
NEW (in introduction):
   This document defines a well-known community value for the purpose of
   reducing the management overhead of gracefully shutting down BGP
   sessions. The well-known community allows implementers to provide an
   automated graceful shutdown mechanism that does not require any
   router reconfiguration at maintenance time.

OLD (in Make before break convergence: g-shut):
   This section describes configurations and actions to be performed for
   the graceful shutdown of eBGP sessions, iBGP sessions or a whole BGP
   speaker.
NEW:
   This section describes configurations and actions to be performed for
   the graceful shutdown of BGP sessions.

OLD (in Pre-configuration):
   On each ASBR supporting the g-shut procedure, an outbound BGP route
   policy is applied on all iBGP sessions of the ASBR, that:
   o  matches the g-shut community
   o  sets the LOCAL_PREF attribute of the paths tagged with the g-shut
      community to a low value
   o  removes the g-shut community from the paths.
   o  optionally, adds an AS specific g-shut community on these paths to
      indicate that these are to be withdrawn soon.  If some ingress
      ASBRs reset the LOCAL_PREF attribute, this AS specific g-shut
      community will be used to override other LOCAL_PREF preference
      changes.
NEW (in Pre-configuration):
   On each ASBR supporting the g-shut procedure, an inbound BGP route
   policy is applied on all BGP sessions of the ASBR, that:
   o  matches the g-shut community
   o  sets the LOCAL_PREF attribute of the paths tagged with the g-shut
      community to a low value (a value of zero is recommended).

OLD (in Operations at maintenance time):
   On the g-shut initiator, upon maintenance time, it is required to:
   o  apply an outbound BGP route policy on the maintained eBGP session
      to tag the paths propagated over the session with the g-shut
      community.  This will trigger the BGP implementation to re-
      advertise all active routes previously advertised, and tag them
      with the g-shut community.
   o  apply an inbound BGP route policy on the maintained eBGP session
      to tag the paths received over the session with the g-shut
      community.
   o  wait for convergence to happen.
   o  perform a BGP session shutdown.
NEW (in Operations at maintenance time):
   On the g-shut initiator, upon maintenance time, it is required to:
   o  apply an outbound BGP route policy on the maintained EBGP session
      to tag the paths propagated over the session with the g-shut
      community. This will trigger the BGP implementation to re-
      advertise all active routes previously advertised, and tag them
      with the g-shut community.
   o  apply an inbound BGP route policy on the maintained EBGP session
      to tag the paths received over the session with the g-shut
      community and set a low LOCAL_PREF value.
   o  wait for convergence to happen.
   o  shutdown the EBGP session, optionally using [RFC-To-Be
      draft-ietf-idr-shutdown] to signal the reason of the shutdown.

OLD (in BGP implementation support for g-Shut):
   1.  On the eBGP side, update all the paths propagated over the
       corresponding eBGP session, tagging the g-shut community to them.
       Any subsequent update sent to the session being gracefully shut
       down would be tagged with the g-shut community.
   2.  On the iBGP side, lower the LOCAL_PREF value of the paths
       received over the eBGP session being shut down, upon their
       propagation over iBGP sessions.  Optionally, also tag these paths
       with an AS specific g-shut community.
NEW (in BGP implementation support for g-Shut):
   1.  On the EBGP side, update all the paths propagated over the
       corresponding eBGP session, tagging the g-shut community to them.
       Any subsequent update sent to the session being gracefully shut
       down would be tagged with the g-shut community.
   2.  lower the LOCAL_PREF value of the paths received over the EBGP
       session being shut down.

OLD:
7.  IANA assigned g-shut BGP community
   Applying the g-shut procedure is rendered much easier with the use of
   a single g-shut BGP community value [RFC1997] which could be used on
   all eBGP sessions, for both inbound and outbound signaling.  The
   community value 0xFFFF0000 has been assigned by IANA for this
   purpose.
8.  IANA Considerations
   This document has no actions for IANA.
NEW:
7.  IANA Considerations
   The IANA has assigned the value 0xFFFF0000 to the planned-shut
   community in the "BGP Well-known Communities" registry. IANA is
   requested to change the name planned-shut to GRACEFUL_SHUTDOWN.
------

Kind regards,

Job

---

From: "Will Hargrave" <[email-masked]>
Date: Sat, 24 Jun 2017 11:45:55 +0100

On 22 Jun 2017, at 21:47, Job Snijders wrote:

>       community to a low value (a value of zero is recommended).

Since we define some terminology in 2., we should use it:

On each g-shut neighbor ASBR, an inbound BGP route policy is applied on 
all BGP sessions of the ASBR, that:

I don’t think the ‘neighbor’ / ‘initiator’ terminology is very 
clear, but I struggle to find more appropriate terms right now. 
‘g-shut receiver’?

>       community and set a low LOCAL_PREF value.

‘Maintained’ (in ‘maintained EBGP session’) is not a useful word 
to use here, since ultimately the session is not to be maintained at 
all! Perhaps “eBGP session” is clear enough, or use ‘impacted’ 
or ‘relevant’.

I have a further comment on section 6. Link Up cases. I rather feel this 
is a separate body of work, and belongs in some sort of ‘gshut-bis’ 
where we can extend on this technique. Certainly there is nothing in the 
rest of the draft which prevents the use of gshut technique to improve 
link-up behaviour, so I would remove sections 6/6.1/6.2 for clarity.

Behaviour in relation to next-hop reachability across a multi-access 
network is of particular interest to me as an IXP operator, and is 
probably a whole topic of its own.

-- 
Will Hargrave
Technical Director
LONAP Ltd
+44 114 303 4444

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Mon, 26 Jun 2017 00:15:38 +0000

I agree with Job's proposals.

In particular, the removal of the g-shut community should be carefully
considered. On the one hand, the g-shut community may not be originated
by the neighbor and is valid wherever the route may be advertised.
On the other hand, in the IPv4 and IPv6 address families, if an alternate
route is not available, the g-shut community can be spread throughout
all routers in the world, causing excessive churn.
I understand that this issue has been discussed in the working group,
but the concerns should be written into the RFC.

I think section 6 for link up should stay.
The cases desctibed are real, whether corner or not.
It does not even need route reflectors to happen.
All that is required is for a withdraw message to be processed by a router
before the new announcement.

Consider:
R2 is announcing the best-path.
R1 was under maintenance and is coming up.
R1 advertises the new path to R2.
R2 withdraws its path.
R1 advertises the new path to R3.
R3 receives or processes the withdrawal from R2 before the advertisement from R1.

"4.2.3 Router g-shut" should make it clear that the behavior is in addition
to the behavior of the links being shut down. Shutting down a router means all its
links are going down too. In addition, the gshut community should be tagged
onto the originated routes.

Thanks,
Jakob.

> -----Original Message-----
> From: GROW [mailto:[email-masked]] On Behalf Of Will Hargrave
> Sent: Saturday, June 24, 2017 3:46 AM
> To: Job Snijders <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [GROW] draft-ietf-grow-bgp-gshut
> 
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: <[email-masked]>
Date: Mon, 26 Jun 2017 13:57:54 +0000

Hi Job,

Thanks for the comments.
Please see inline [Bruno]

 > Sent: Thursday, June 22, 2017 10:47 PM
> 
 > Dear working group,
 > 

[Bruno] Will answer these technical points in separate threads.

 > supplied, if there is interest in this I am happy to provide that.
 
[Bruno] I think this would be useful. (although CLI change over time)
 
 >    initiated for the tagged prefix.
 
[Bruno] OK. Slightly reworded as "It defines a well-known BGP community, called gshut, to signal the graceful shutdown of paths to other Autonomous Systems."
 
 >    router reconfiguration at maintenance time.

[Bruno] OK.

 >    the graceful shutdown of BGP sessions.
 
[Bruno] OK.

 >       community to a low value (a value of zero is recommended).
 
[Bruno] Change delayed, as related to the 2 technical points to be discussed in separate threads.
 
 >       draft-ietf-idr-shutdown] to signal the reason of the shutdown.
 
[Bruno] Change delayed, as related to the 2 technical points to be discussed in separate threads.
Added text related to ID-shutdown
 
 >    2.  lower the LOCAL_PREF value of the paths received over the EBGP
 
[Bruno] Change delayed, as related to the 2 technical points to be discussed in separate threads.
 
 >    requested to change the name planned-shut to GRACEFUL_SHUTDOWN.

[Bruno] OK. except that in this document, the name "g-shut" is used. (rather than GRACEFUL_SHUTDOWN). So using the name g-shut.

Kind regards,
--Bruno

---

From: <[email-masked]>
Date: Mon, 26 Jun 2017 13:58:23 +0000

Hi Will,

Thanks for the comments.

 > Sent: Saturday, June 24, 2017 12:46 PM
> 
 
[Bruno] ok. " g-shut neighbor ASBR" feels not so easy to read. Proposing
OLD: On each ASBR supporting the g-shut procedure,
NEW: On each ASBR supporting the g-shut neighbor procedure,

 > clear, but I struggle to find more appropriate terms right now.

[Bruno] would work for me as this is close to the "Graceful Restart" terminology using Restarting Speaker / Receiving Speaker.
Would also require changing the same of the neighbor AS:
OLD: Neighbor AS: the Autonomous System of the g-shut neighbor.
NEW: Receiver AS: the Autonomous System of the g-shut receiver.

Waiting for some more feedback before applying the change.

 > all! Perhaps “eBGP session” is clear enough, or use ‘impacted’
 
[Bruno] would the below change work for you?
OLD: apply an outbound BGP route policy on the maintained eBGP session to
       tag the paths propagated over the session with the g-shut community.
NEW: apply an outbound BGP route policy on the eBGP session to be shutdown. This policy tags
       the paths propagated over the session with the g-shut community.

 > link-up behaviour, so I would remove sections 6/6.1/6.2 for clarity.

[Bruno] I feel that this is useful to state that the establishment of an EBGP session can trigger loss of connectivity, as I don't feel that this is obvious to everyone, nor intuitive. (this is supposed to be a "good news").
In term of draft clarity, this draft mix both an operational procedure and discussion related to BGP convergence. Possibly the draft could be reorganized to better identify the operational procedure (currenly in §4.2.1 "EBGP g-shut". I'm considering the following change

OLD:
   4.  Practices to avoid packet losses
     4.1.  Improving availability of alternate paths
     4.2.  Make before break convergence: g-shut 
   5.  Forwarding modes and transient forwarding loops during convergence
   6.  Link Up cases 
     6.1.  Unreachability local to the ASBR  
     6.2.  iBGP convergence  

NEW:
   4.  Practices to avoid packet losses
     4.1.  Improving availability of alternate paths
     4.2.  Make before break convergence: g-shut [4.2 introduction only]
     4.3  Forwarding modes and transient forwarding loops during convergence
  5 EBGP g-shut procedure
    5.1 pre-configuration
    5.2 operation at maintenance time
    5.4 BGP implementation support for g-shut
   6.  Other cases
    6.1 IBGP g-shut
    6.2 Link Up cases 

Thanks again for the comments,
Regards,
--Bruno

---

From: <[email-masked]>
Date: Mon, 26 Jun 2017 13:58:32 +0000

Hi Jakob,

Thanks for the comments.

> From: GROW [mailto:[email-masked]] On Behalf Of Jakob Heitz (jheitz)
 > Sent: Monday, June 26, 2017 2:16 AM
> 

[Bruno] I'll initiate a thread to rediscuss this point. Then we'll see the outcome to update the draft.
 
 > The cases desctibed are real, whether corner or not.

[Bruno] agreed.

 > R3 receives or processes the withdrawal from R2 before the advertisement from R1.
 
[Bruno] Indeed. In the draft, we presented the IBGP Route Reflector case, as we though that RR is more common than full mesh.
Note that both cases are essentially the same:
- IBGP full mesh between ASBR in your example
- IBGP full mesh between RR in the draft
 
In both cases, we have:
- a distributed convergence with 2 nodes involved,
- multiple/many IBGP sessions, creating multiple signaling paths, where a BGP UPDATE flowing along a 'long' path (2 sessions) may arrive after a BGP session flowing along a 'short' path (1 session).

(Unfortunately) I agree that this is not theoretical.

 > onto the originated routes.
 
[Bruno] Agreed. Proposed text:
OLD:
   In the case of a shutdown of a router, a reconfiguration of the
   outbound BGP route policies of the g-shut initiator SHOULD be
   performed to set a low LOCAL_PREF value for the paths originated by
   the g-shut initiator (e.g, BGP aggregates redistributed from other
   protocols, including static routes).

   This behavior is equivalent to the recommended behavior for paths
   "redistributed" from eBGP sessions to iBGP sessions in the case of
   the shutdown of an ASBR.

NEW:
In the case of a shutdown of the whole router, in addition to the g-shut of all EBGP sessions, there is a need to g-shut the routes originated by this router (e.g, BGP aggregates redistributed from other protocols, including static routes).  This can be performed by tagging such routes with the g-shut community.

Thanks,
Regards,
--Bruno

---

From: <[email-masked]>
Date: Mon, 26 Jun 2017 13:58:40 +0000

Hi Job, all

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, June 22, 2017 10:47 PM
[...]

> I think that the neighbor ASBR should _not_ strip the GSHUT well-known community.

I'm personally open to both options.
Discussing this further:

- First of all, stripping the g-shut well-known community fits the goal of the document and its requirements (RFC 6198). The main motivation is to have a g-shut solution with no required BGP protocol extension, and where the backup path is known to the AS. i.e. we are not looking for an Internet wide convergence/g-shut. We are primarily interested in a g-shut within the responsibility of the AS. IOW, when it's "my fault" if my customer experiences a packet loss. In this case, there is no need to propagate the g-shut community further.

- removing the g-shut community reduces Internet wide churns, including compared to when g-shut is not used.
Here are the 3 cases, focusing on the first step of the BGP convergence (searching for the backup path)
  - No g-shut: a WITHDRAW is propagated to other ASes
  - g-shut propagating the community: an UDPDATE is propagated to others ASes (same path, adding the community)
  - g-shut removing the community: no BGP messages propagated to others ASes (same path, same communities/attribute)

In general, reducing the BGP churn is considered as a feature. By removing the g-shut community, we are at the same time:
a) g-shut in both affected ASes
b) reducing Internet churn.

Now if we choose to not remove the community, we improve "a" by covering the cases where the backup path is unknown to the AS. And we keep "b" unchanged compared to today (but degrade it compare to current g-shut draft).
As for my requirements, I'm considering that our ASes have the knowledge of the backup path. Hence I don't need for the extra coverage. Regarding the extra cost, I agree that one can hardly consider this unacceptable since this is the current behavior.

TL;DR: it's a tradeoff between 2 secondary objectives:
- reducing Internet churned (compared to today)
- improving the g-shut coverage when the AS does not know the backup path

Draft can possibly discuss both options, at the cost of additional reading complexity. But this possibly could be discussed in a different section.
I'd welcome more opinion, before choosing the main text.

Thanks,
Regards,
--Bruno

---

From: <[email-masked]>
Date: Mon, 26 Jun 2017 14:14:19 +0000

Hi Job ,all

[...] 

> the place where the low local preference is set

Current version of the draft changes the LOCAL_PREF on the Adj-RIB-Out and not on the Loc_RIB.
This has a technical benefit as otherwise, the router may (locally) select a backup route which it is not allowed to advertise, which would trigger the sending of a withdraw of the original route. i.e. not a g-shut.
This may be considered as a corner case, but I don't see why we would choose not cover it.
I suppose that one can also see some operational drawbacks:
a) seems less intuitive.
b) traffic received from external interfaces on this specific g-shut router (the egress in the AS) are forwarded on the "nominal"/g-shut path, rather than on the backup path.

IMO:
- "a" is not a big concern as the related configuration is pre-configured once for all on the router. We are not discussing the configuration applied at maintenance time.
- "b" has no impact on the customer loss of connectivity as this traffic may be locally rerouted in no time (up to zero packet loss) by the router which needs to update its FIB "in place". i.e. the destination IP prefix is never removed from the FIB. Only the outgoing interface is changed, and during this change, both outgoing interfaces are valid (both the old and the new).

So although some may see a tradeoff, I'd rather favor the generality of the mechanism. Specifically as not covering the corner cases may surprise the operator.

Thanks,
Regards,
--Bruno

---

From: Job Snijders <[email-masked]>
Date: Mon, 26 Jun 2017 17:30:44 +0200

> Current version of the draft changes the LOCAL_PREF on the Adj-RIB-Out

Yes, that should change.

> This has a technical benefit as otherwise, the router may (locally)
> select a backup route which it is not allowed to advertise, which
> would trigger the sending of a withdraw of the original route. i.e.
> not a g-shut.

Sure, but the reverse is true too. It may select a path that it is not
allowed to advertise and with the gshut community select something else.
This is not a strong argument.

> b) traffic received from external interfaces on this specific g-shut
> router (the egress in the AS) are forwarded on the "nominal"/g-shut
> path, rather than on the backup path.

Yes, and avoiding the nominal path (using a backup path) has great
benefit that the operator can monitor whether convergence is finished,
which can be used in the decision making process when to proceed with
the maintenance.

Operational expectation is that when one initiates a process to drain
traffic from a node or a link, that traffic is actually, visibly drained
away from a link.

> IMO:
> - "a" is not a big concern as the related configuration is
> pre-configured once for all on the router. We are not discussing the
> configuration applied at maintenance time.
> - "b" has no impact on the customer loss of connectivity as this
> traffic may be locally rerouted in no time (up to zero packet loss) by
> the router which needs to update its FIB "in place". i.e. the
> destination IP prefix is never removed from the FIB. Only the outgoing
> interface is changed, and during this change, both outgoing interfaces
> are valid (both the old and the new).

This assumes that all parties involved can actually locally reroute
without packetloss. One cannot know this. Also, what of the case where a
single ASN is composed of a single router (or a network is operated
without the concept of IBGP as defined by IETF).

> So although some may see a tradeoff, I'd rather favor the generality
> of the mechanism. Specifically as not covering the corner cases may
> surprise the operator.

Also, as it currently stands operators, are implementing it on Loc-RIB.

Another argument in favor of adjustment on Loc-RIB rather then "On
Adj-RIB-Out but only for IBGP sessions", is that it is much easier to
explain to people: "When you attach the GRACEFUL_SHUTDOWN, everyone is
expected to lower LOCAL_PREF as low as possible" (and simply forgo
explaining the particular conditionals of the current draft).

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Mon, 26 Jun 2017 17:39:12 +0200

On Mon, Jun 26, 2017 at 01:58:40PM +0000, [email-masked] wrote:
> - First of all, stripping the g-shut well-known community fits the
> goal of the document and its requirements (RFC 6198). The main
> motivation is to have a g-shut solution with no required BGP protocol
> extension, and where the backup path is known to the AS. i.e. we are
> not looking for an Internet wide convergence/g-shut. We are primarily
> interested in a g-shut within the responsibility of the AS. IOW, when
> it's "my fault" if my customer experiences a packet loss. In this
> case, there is no need to propagate the g-shut community further.
> 
> - removing the g-shut community reduces Internet wide churns,
> including compared to when g-shut is not used.
> Here are the 3 cases, focusing on the first step of the BGP
> convergence (searching for the backup path)
>   - g-shut propagating the community: an UDPDATE is propagated to
>   others ASes (same path, adding the community)
>   - g-shut removing the community: no BGP messages propagated to
>   others ASes (same path, same communities/attribute)
> 
> In general, reducing the BGP churn is considered as a feature. By
> removing the g-shut community, we are at the same time:
> Now if we choose to not remove the community, we improve "a" by
> covering the cases where the backup path is unknown to the AS. And we
> keep "b" unchanged compared to today (but degrade it compare to
> current g-shut draft).

I don't think we really have a say in whether BGP Communities are
removed or not. This is outside our control. If we remove the hard
requirement to remove the community, the behaviour falls back to
https://tools.ietf.org/html/rfc7454#section-11 which imho is fine: leave
the choice with the operator. Perhaps the draft does not need to define
whether to strip or not.

I have to admit that the first time I used a a 'g-shut -06 compliant'
automated process, I was surprised that that I didn't see the community
on the IBGP outbound announcements. Removing the community somewhat
reduces visibility to those involved with the operation.

> As for my requirements, I'm considering that our ASes have the
> knowledge of the backup path. Hence I don't need for the extra
> coverage. Regarding the extra cost, I agree that one can hardly
> consider this unacceptable since this is the current behavior.
> 

+ improve visibility into the operation

> Draft can possibly discuss both options, at the cost of additional
> reading complexity. But this possibly could be discussed in a
> different section.
>

Bruno, perhaps my interpretation of the above sentence's phrasing is
somewhat mangled because neither of us are a native speaker, but keep in
mind this is a GROW working group document. :-)

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Mon, 26 Jun 2017 17:43:00 +0200

>  > The cases desctibed are real, whether corner or not.

I agree it is valuable text, but the draft is about "SHUTDOWN" - not
about "Link Up". Perhaps spin it off into a more complete draft
focussing specifically on this topic?

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Mon, 26 Jun 2017 17:50:05 +0200

On Mon, Jun 26, 2017 at 01:57:54PM +0000, [email-masked] wrote:
> [Bruno] OK. except that in this document, the name "g-shut" is used.
> (rather than GRACEFUL_SHUTDOWN). So using the name g-shut.

I think "g-shut" should be changed through the document to
GRACEFUL_SHUTDOWN. If we study the registry at 
https://www.iana.org/assignments/bgp-well-known-communities/ it is clear
that the convention is to use uppercase and a _ (underscore) separator.

I also do not see a compelling reason to use an abbreviation, as the
tradeful is saving a few bytes vs readability for novice users.

The concept described in the document is called "graceful shutdown", not
"g-shut", the community name should simply reflect that. Changing the
name through the document will help popularise the idea, since then the
name of the concept and the name of the community are aligned with each
other letter wise.

RTGDIR / OPSDIR / IESG are likely to be picky about it anyway, better to
preempt that. :)

Kind regards,

Job

ps. The "ACCEPT-OWN" community seems to be mistakingly listed the way it
is, and should be "ACCEPT_OWN" as that is what RFC 7611 requested from
IANA. I've notified the relevant AD.

---

From: heasley <[email-masked]>
Date: Mon, 26 Jun 2017 17:06:58 +0000

> [Bruno] OK. Slightly reworded as "It defines a well-known BGP community, called gshut, to signal the graceful shutdown of paths to other Autonomous Systems."

s/paths to other Autonomous Systems./a path in the presence of other paths./

ie: it does nothing when there is no other path and it may not move to
anoter AS, just to another path.  it is per-session, not per-AS.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Mon, 26 Jun 2017 17:55:41 +0000

It works for iBGP links too. An iBGP link is not to another AS.

Thanks,
Jakob.

> From: GROW [mailto:[email-masked]] On Behalf Of heasley
> Sent: Monday, June 26, 2017 10:07 AM
> To: [email-masked]
> Cc: [email-masked]
> > [Bruno] OK. Slightly reworded as "It defines a well-known BGP community, called gshut, to signal the graceful
> shutdown of paths to other Autonomous Systems."
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Mon, 26 Jun 2017 20:50:57 +0000

Local pref should be reduced when the route is received.

Consider router that learns a route on 2 interfaces.
The best path is on the interface that will go down.
The right thing to do is to change the best path to the other one.
Lowering local pref at the incomming interface will do that.

Thanks,
Jakob.

> From: GROW [mailto:[email-masked]] On Behalf Of Job Snijders
> Sent: Monday, June 26, 2017 8:31 AM

---

From: Ben Maddison <[email-masked]>
Date: Tue, 27 Jun 2017 05:52:25 +0000

Hi all,

Comments in-line [BM]

On Mon, 2017-06-26 at 17:30 +0200, Job Snijders wrote:
> >  > Sent: Thursday, June 22, 2017 10:47 PM
> > 
> > [...] 
> > 
> >  > should move closer to the initiator of the gshut.  Instead of
> > setting
> >  > the low LP on Adj-RIB-Out to IBGP neighbors, the low LP should
> > be set
> >  > during application of the local policy on the relevant Adj-RIB-
> > In.
> > 
> > Current version of the draft changes the LOCAL_PREF on the Adj-RIB-
> > Out
> > and not on the Loc_RIB.
> 
> Yes, that should change.

[BM] Agreed. The semantics of the community are: "this path will go
away soon, please stop using it". This hint seems to me to be
applicable to any receiving speaker.

> This is not a strong argument.

[BM] Additionally, other things being equal, surely that withdraw is
going to be sent at shutdown time anyway? Triggering it early seems to
be harmless at worst.

> without the concept of IBGP as defined by IETF).

[BM] We should also consider the outcome of the parallel discussion
regarding the removal of the gshut community (for the record, i'm in
favour of propagating by default and putting the operator in charge of
overriding that behaviour).
In the extreme case where all adjacencies are EBGP, if community
propagates everywhere, but no speaker ever acts on it, you get all the
churn for none of the benefit.
> explaining the particular conditionals of the current draft).
[BM] Also agreed. Although calling out corner cases that may warrant
attention is also valuable for those deploying new features. For me,
these should live in a "deployment considerations" section, if at all.

Cheers,

Ben

-- 
Ben Maddison

Director
Workonline Communications (Pty) Ltd
 
Office: +27 (0) 21 200 9000
Mobile: +27 (0) 82 415 5545
Email:  [email-masked]
SIP:    [email-masked]

---

From: Ben Maddison <[email-masked]>
Date: Tue, 27 Jun 2017 06:31:11 +0000

Hi all,

In-line as [BM]

> > case, there is no need to propagate the g-shut community further.

[BM] However, if by propagating the community we get reasonable
Internet-wide g-shut coverage, then why prevent that?

> >   others ASes (same path, adding the community)

[BM] This generates an "interim" UPDATE (adding the community) only
where the propagating AS has no alternate path. In that case, the
additional UPDATE provides the remote AS an opportunity to re-converge
around the shutdown at the AS-graph level, and therefore seems to me to
have value.
Alternatively, if the receiving AS does have an alternate path then:
a) the alternate path is eligible for export, then an UPDATE may be
generated if the alt-path has differing attributes. This UPDATE would
eventually be generated at shutdown-time anyway, so no additional churn
here; or
b) the alternative path is not eligible for export, then a WITHDRAW is
generated, which would be generated eventually at shutdown-time anyway,
so no additional churn here either.

> >   others ASes (same path, same communities/attribute)

[BM] Correct, but also no opportunity to reroute traffic around the AS
that will shortly be sending a WITHDRAW anyway.

> the choice with the operator. Perhaps the draft does not need to

[BM] Agreed. I don't see a case for treating this community differently
from any other 1997 community in this respect.

> + improve visibility into the operation

[BM] In particular, a 3rd party that is trying to self-service an
answer to the question "why did the traffic move" will be looking at a
looking-glass rather than a router CLI. More visibility is particularly
helpful in this case.

Cheers,

Ben

-- 
Ben Maddison

Director

---

From: <[email-masked]>
Date: Tue, 27 Jun 2017 13:14:34 +0000

> From: heasley [mailto:[email-masked]]  > Sent: Monday, June 26, 2017 7:07 PM
 > To: DECRAENE Bruno IMT/OLN
 > Cc: [email-masked]
 > > [Bruno] OK. Slightly reworded as "It defines a well-known BGP community, called gshut, to
 > signal the graceful shutdown of paths to other Autonomous Systems."
 > 

My sentence was misleading. Changing to: "It defines a well-known BGP community, called g-shut, to signal over the EBGP session to be shutdown, the graceful shutdown of paths."

Thanks,
--Bruno

---

From: <[email-masked]>
Date: Tue, 27 Jun 2017 13:14:53 +0000

Job,

> From: Job Snijders [mailto:[email-masked]]  > Sent: Monday, June 26, 2017 5:39 PM
> 

To rephrase: I'd welcome more feedback from the GROW WG before reflecting the consensus in the document.

--Bruno

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 28 Jun 2017 05:52:42 +0000

Bruno,

To my mind, the purpose of graceful shutdown is to tease out the
hidden paths before sending the withdraw. In your cases, the
alternative paths are not hidden. They are already available.
If they are available to the gshut initiating router, then they
are available to the other routers. Therefore a withdraw is ok.

Now, consider a not uncommon case:
The gshut initiating router has 2 advertiseable paths.
Another router has an inferior path.
When the gshut is initiated for one path, the inferior path
from the other router will be chosen by the network.
When gshut is removed, the network will churn a second time
when it chooses the second path.

I think it's better for the gshut initiating router to just
advertise its next best path instead of gshut.

Thanks,
Jakob.

> From: GROW [mailto:[email-masked]] On Behalf Of [email-masked]
> Sent: Tuesday, June 27, 2017 6:15 AM
> I'm not following you, so I'll provide an example of "not allowed to be advertised on IBGP"
> Topology:  the g-shut initiator has an IBGP session with another ASBR (e.g. ASBR1 in the same POP). This ASBR knows
> an alternate path and advertise it over IBGP. (e.g. best external, or EBGP>IBGP)
> g-shut convergence: If the g-shut initiator applies the low local_pref, it will switch to this alternate path. As it
> can't advertise over IBGP a path learned over IBGP, it will send a withdraw to both its Route Reflector and its own
> IBGP peer.
> Discussion: May be we could assume that its Route Reflector and IBGP peers also have the pre-knowledge of this
> alternate path (from ASBR1) in which case this is not a big deal. But in theory, it's just safer to never send a
> withdraw.

---

From: <[email-masked]>
Date: Wed, 28 Jun 2017 09:17:16 +0000

Jakob,

> From: Jakob Heitz (jheitz) [mailto:[email-masked]]
 > Sent: Wednesday, June 28, 2017 7:53 AM
> 
 > Bruno,
 > 

The purpose of the graceful shutdown is to reduce, possibly remove, the loss of connectivity when an EBGP session is shutdown.
The main contribution is indeed the time taken to learn and install the backup paths.
What is needed is an AS global convergence, not just the g-shut initiator.
In this specific case, the alternative path is known to the g-shut initiator. That does not mean that it is known by all routers in the AS.

 > are available to the other routers. 

Why?

> Therefore a withdraw is ok.

If we want to avoid any node to have a lack of path, avoiding withdraw is just safer.
Especially considering the wide possibilities of BGP topologies inside the AS.
  
 > when it chooses the second path.
 
Sure.
>From a probability standpoint, there is more chance that the post convergence path be advertised by one of the other (e.g. 100) routers than by this (ingle) g-shut router. Especially assuming that most AS would want to protect against node failure, the backup path would typically be connected on a different node (if available of course).
Note also that your "second churn" is part of IBGP path hunting and may happen in any case (i.e. the first alternate path found may not the best/post convergence one)
 
 > advertise its next best path instead of gshut.
 
OK. Thanks for the feedback.
This seems to be the general sense of the GROW WG. I'll wait some more time to have a chance to collect all feedbacks, and we'll update the draft.
 
Thanks
--Bruno

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 28 Jun 2017 20:12:56 +0000

Bruno,

>  > are available to the other routers.

The advertising router advertised it.
Your example is iBGP. When one speaker advertises, the whole AS receives it.
Now suppose because of some weirdness, not every speaker in the AS receives it.
Then even a gshut community isn't going to make a difference.
Even after the gshut, this iBGP route isn't going to get any further.

Thanks,
Jakob.

---

From: <[email-masked]>
Date: Wed, 28 Jun 2017 20:55:32 +0000

Jakob,

 > Sent: Wednesday, June 28, 2017 10:13 PM
> 
 > Bruno,
 > 

If you assume an IBGP full mesh, I agree.
If you have a Route Reflector topology, without add-path, I disagree.
Only one path is advertised by the RR. We can always find a node A, in the Initiator AS, which uses the g-shut initiator as best path/Next Hop (otherwise, nobody uses the path advertised by the g-shut initiator, and there is no need for g-shut). That node A received a single path from its RR (the path from the g-shut initiator), hence it does not know an alternate path. Even if another node B advertises an alternate path thanks to best-external or by preferring its EBGP path over an IBGP path (both paths having the same LOCAL_PREF, aspath length, origin, MED)

 > Even after the gshut, this iBGP route isn't going to get any further.

I'm not sure to follow your point.
The route being gshut has a low local_pref hence the alternate route with become best and propagated across the AS.

Thanks
--Bruno

---

From: heasley <[email-masked]>
Date: Wed, 28 Jun 2017 20:58:16 +0000

> + improve visibility into the operation

+1 for that; and allows it to (possibly) enter the records of routeviews.

& allows other ASes beyond the receiver to take action - even if the receiver
  does not have an alternate path ... which imiho ends this discussion about
  removing the community.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 28 Jun 2017 21:51:31 +0000

You're right Bruno. I misstated it.

Still, node A will ever have no path available.
Whether Gshut initiator sends gshut or withdraws, the result is the same:
RR sends the new path to Node A.

Thanks,
Jakob.

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, June 28, 2017 1:56 PM
> To: Jakob Heitz (jheitz) <[email-masked]>
> Cc: [email-masked]
> Subject: RE: [GROW] draft-ietf-grow-bgp-gshut
> 
> Jakob,
> 
> 
> Only one path is advertised by the RR. We can always find a node A, in the Initiator AS, which uses the g-shut
> initiator as best path/Next Hop (otherwise, nobody uses the path advertised by the g-shut initiator, and there is no
> need for g-shut). That node A received a single path from its RR (the path from the g-shut initiator), hence it does
> not know an alternate path. Even if another node B advertises an alternate path thanks to best-external or by
> preferring its EBGP path over an IBGP path (both paths having the same LOCAL_PREF, aspath length, origin, MED)
> 
>

---

From: heasley <[email-masked]>
Date: Thu, 29 Jun 2017 18:27:46 +0000

> My sentence was misleading. Changing to: "It defines a well-known BGP community, called g-shut, to signal over the EBGP session to be shutdown, the graceful shutdown of paths."

As Jakob mentioned, its not ebgp-only, as it also addresses shutting an
ibgp or an entire speaker.

---

From: <[email-masked]>
Date: Fri, 30 Jun 2017 15:26:08 +0000

> From: heasley [mailto:[email-masked]]  > Sent: Thursday, June 29, 2017 8:28 PM
> 
 > > My sentence was misleading. Changing to: "It defines a well-known BGP community,
 > called g-shut, to signal over the EBGP session to be shutdown, the graceful shutdown of
 > paths."
 > 

The procedure is not limited to ebgp. However the BGP community is defined in order to perform the signaling specifically over EBGP.
Over IBGP, we use LOCAL_PREF to lower the preference of the route. Even if the community may be kept for information.

Thanks,
Regards,
--Bruno

---

From: <[email-masked]>
Date: Fri, 30 Jun 2017 15:42:55 +0000

> From: Jakob Heitz (jheitz) [mailto:[email-masked]]  > Sent: Wednesday, June 28, 2017 11:52 PM
> 

In the end, yes.
But if the gshut initiator sends a withdraw to its RR, and its RR does not have the pre-knowledge of a backup path, the withdraw is first propagated to the RR clients.

However this is a corner case as it would assume the gshut initiator to know more IBGP paths than its RR.

Given that most people would prefer applying the low LOCAL_PREF on the gshut initiator itself, let's do this. Hopefully I'll upload a new version before the deadline (Monday)

Thanks for the feedback
--Bruno

 > _________________________________________________________________________
 > ___________________________________________
 > > _____
 > >
 > > Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou
 > privilegiees et ne doivent
 > > donc
 > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par
 > erreur, veuillez le
 > > signaler
 > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant
 > susceptibles
 > > d'alteration,

---

From: heasley <[email-masked]>
Date: Fri, 30 Jun 2017 18:18:56 +0000

> Over IBGP, we use LOCAL_PREF to lower the preference of the route. Even if the community may be kept for information.

you would still want to add the community; if you have no other path, you
still want the community to be on routes to remote eBGP peers, if you have
any in-bound iBGP policy that might affect the LP, you still want/need to
communicate to the ibgp peers.