Subject: [Idr] IETF LC for IDR-ish document <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation Behavior Without Policies) to Proposed Standard

From: "John G. Scudder" <[email-masked]>
Date: Wed, 19 Apr 2017 12:49:42 -0400

IDR folks,

As many of you have already noticed, draft-ietf-grow-bgp-reject-05 has completed GROW WGLC and is now in IETF LC.

As nobody other than Alvaro noticed (thank you for noticing, Alvaro!) draft-ietf-grow-bgp-reject-05 represents an update to RFC 4271, in that it mandates what a BGP implementation MUST do. See section 2 of the draft for the details. It's short and easy to read.

If we had noticed this earlier, we would have either chosen to home the document in IDR, or explicitly made an exception to have GROW do the work. Given that we didn't, though, the plan is to continue progressing the draft as a GROW document. However:

- As I understand it, the authors will add the Updates: 4271 header in addition to potentially taking in other comments from AD review.
- If anyone has a strong objection to the unusual procedure, please say so (either on-list, or to the chairs + AD).
- Please send any last call comments to the IETF LC (see below) although it's also OK to discuss here on the IDR list of course.

Many IDR participants are also active in GROW and have had their say, but if you haven't, now's your chance.

Thanks,

--John

> Begin forwarded message:
> 
> From: The IESG <[email-masked]>
> Subject: Last Call: <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation Behavior Without Policies) to Proposed Standard
> Date: April 18, 2017 at 5:16:05 PM EDT
> To: "IETF-Announce" <[email-masked]>
> Cc: [email-masked], [email-masked], [email-masked], [email-masked]
> Reply-To: [email-masked]
> 
> 
> The IESG has received a request from the Global Routing Operations WG
> (grow) to consider the following document:
> - 'Default EBGP Route Propagation Behavior Without Policies'
> <draft-ietf-grow-bgp-reject-05.txt> as Proposed Standard
> 
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2017-05-02. Exceptionally, comments may be
> sent to [email-masked] instead. In either case, please retain the
> beginning of the Subject line to allow automated sorting.
> 
> Abstract
> 
>  This document defines the default behavior of a BGP speaker when
>  there is no import or export policy associated with an External BGP
>  session.
> 
> 
> The file can be obtained via
> https://datatracker.ietf.org/doc/draft-ietf-grow-bgp-reject/
> 
> IESG discussion can be tracked via
> https://datatracker.ietf.org/doc/draft-ietf-grow-bgp-reject/ballot/
> 
> This IETF LC, which originally concluded on 2017-04-18, is being 
> extended to allow for additional input to be provided. Ops AD (for GROW) 
> and Routing AD (for IDR) wish to ensure that cross WG discussions have 
> had a chance to occur.
> 
> No IPR declarations have been submitted directly on this I-D.

---

From: Keyur Patel <[email-masked]>
Date: Wed, 19 Apr 2017 20:43:33 +0000

Thank you John for bringing it on IDR. 

As an update to RFC4271, I am not sure if I agree with the EBGP policy configuration. There are lot of DC networks (for example) that use EBGP within their CLOS. This extension may not be applicable in such networks.

I would request authors to consider refining text to include appropriate EBGP use cases and not make it generic for EBGP sessions (defined in 4271).

Regards,
Keyur

On 4/19/17, 9:49 AM, "Idr on behalf of John G. Scudder" <[email-masked] on behalf of [email-masked]> wrote:

    IDR folks,

    

    
    Thanks,
    
    --John
    
    > No IPR declarations have been submitted directly on this I-D.
    
    _______________________________________________
    Idr mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 19 Apr 2017 22:58:41 +0200

As discussed offline with authors I would also suggest to reconsider if
such Standards Track RFC should really be applicable to all AFIs and SAFIs
as it is today. I would suggest to keep it only applicable to 1/1 and 2/1.

For other AFI/SAFIs it will result in empty policy as it is hard to predict
what filters flow-spec or RT constrained (as just two examples) may send
you ahead of time.

For other SAFIs say safi 128 for Inter-AS EBGP the required filtering may
already be present in forms of rt-table-map filtering which would not
necessarily be part of per neighbor eBGP inbound import policy but may
exist depending on the option either globally or per VRF.

Last the draft does not specify if BGP Origin Validation is to be
considered as such inbound policy or not.

Kind regards,
Robert.

On Wed, Apr 19, 2017 at 10:43 PM, Keyur Patel <[email-masked]> wrote:

> As an update to RFC4271, I am not sure if I agree with the EBGP policy
> configuration. There are lot of DC networks (for example) that use EBGP
> within their CLOS. This extension may not be applicable in such networks.
>
> I would request authors to consider refining text to include appropriate
> EBGP use cases and not make it generic for EBGP sessions (defined in 4271).
>
> Regards,
> Keyur
>
>
> On 4/19/17, 9:49 AM, "Idr on behalf of John G. Scudder" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>     IDR folks,
>
>     As many of you have already noticed, draft-ietf-grow-bgp-reject-05 has
> completed GROW WGLC and is now in IETF LC.
>
>     As nobody other than Alvaro noticed (thank you for noticing, Alvaro!)
> draft-ietf-grow-bgp-reject-05 represents an update to RFC 4271, in that it
> mandates what a BGP implementation MUST do. See section 2 of the draft for
> the details. It's short and easy to read.
>
>     If we had noticed this earlier, we would have either chosen to home
> the document in IDR, or explicitly made an exception to have GROW do the
> work. Given that we didn't, though, the plan is to continue progressing the
> draft as a GROW document. However:
>
>     - As I understand it, the authors will add the Updates: 4271 header in
> addition to potentially taking in other comments from AD review.
>     - If anyone has a strong objection to the unusual procedure, please
> say so (either on-list, or to the chairs + AD).
>     - Please send any last call comments to the IETF LC (see below)
> although it's also OK to discuss here on the IDR list of course.
>
>     > Subject: Last Call: <draft-ietf-grow-bgp-reject-05.txt> (Default
> EBGP Route Propagation Behavior Without Policies) to Proposed Standard

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Wed, 19 Apr 2017 20:58:50 +0000

I would agree with Keyur, For better or worse, our Cisco NX-OS BGP
implementation does not require configuration of a peer policy.

In fact, this requirement is contrary to some of the auto-discovery
mechanisms we are exploring where only knowledge of the mutual address
families is required.

Thanks,
Acee 

On 4/19/17, 4:43 PM, "Idr on behalf of Keyur Patel" <[email-masked]
on behalf of [email-masked]> wrote:

>On 4/19/17, 9:49 AM, "Idr on behalf of John G. Scudder"
><[email-masked] on behalf of [email-masked]> wrote:
>
>    IDR folks,
>    
>    As many of you have already noticed, draft-ietf-grow-bgp-reject-05
>has completed GROW WGLC and is now in IETF LC.
>    
>draft-ietf-grow-bgp-reject-05 represents an update to RFC 4271, in that
>it mandates what a BGP implementation MUST do. See section 2 of the draft
>for the details. It's short and easy to read.
>    
>work. Given that we didn't, though, the plan is to continue progressing
>the draft as a GROW document. However:
>    
>    - As I understand it, the authors will add the Updates: 4271 header
>in addition to potentially taking in other comments from AD review.
>    > Cc: [email-masked], [email-masked],
>[email-masked], [email-masked]
>    > Reply-To: [email-masked]
>    > 
>    >

---

From: Jared Mauch <[email-masked]>
Date: Wed, 19 Apr 2017 17:08:43 -0400

If someone sets insecure mode they can  e as promiscuous as they want.  

That mode can have a very low bar IMO. 

Jared Mauch

> On Apr 19, 2017, at 4:58 PM, Acee Lindem (acee) <[email-masked]> wrote:
>

---

From: Gert Doering <[email-masked]>
Date: Wed, 19 Apr 2017 23:18:01 +0200

Hi,

> That mode can have a very low bar IMO. 

This.

If you want "permit any in, any out", nothing in this draft prevents
doing so - but the *default* needs to be "nothing in, nothing out,
unless at least one switch is turned to change that" (= policy configured,
or "I want this to be open" configured).

Gert Doering
        -- Operator, occasional BGP trainer, network hygiene preacher
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: <[email-masked]>
Date: Wed, 19 Apr 2017 22:12:48 +0000

Thanks John for bringing this in IDR.

I admit that I was not following this subject so my comments might be redundant or even stupid.

1) Am I missing something or would this break all existing EBGP sessions deployed with no policy?
If so this would definitely not be deployment friendly. Especially for BGP/MPLS VPN networks using EBGP for PE-CE routing and which have little use of filtering policies.

2) BTW, what is exactly eligible as an "import policy"? e.g.
- is an explicit policy capping the number of received routes eligible as an "import policy"? 
- is Route Target filtering (either automatic or manual) a routing policy?

Same question of "export policy". e.g.
Is an expert policy tagging community eligible?

3) From the introduction
"   There are BGP routing security issues that need to be addressed to
   make the Internet more stable. [...]  This document provides guidance to BGP [RFC4271]
   implementers to improve the default level of Internet routing
   security."

Does this mean that this proposition should be restricted to Internet routing? (while BGP is used for many others applications)

4) Alternatively, there could be a (capability) signaling during the OPEN of the EBGP session. With one end requesting this behavior to its peer. (or alternatively it's peer advertising the presence of a policy and the receiver taking its own decision).

Possibly, some of the requirements may already be addressed by configuring a policy limiting the number of routes acceptable from a peer/customers, and closing the EBGP sessions when this limit is reached. This seems this would catch customers/peers advertising the full routing by mistake/misconfiguration.

Thanks
Regards,
--Bruno

> From John G. Scudder  > Sent: Wednesday, April 19, 2017 6:50 PM
> 
 > IDR folks,
 > 
 > As nobody other than Alvaro noticed (thank you for noticing, Alvaro!) draft-ietf-grow-bgp-
 > reject-05 represents an update to RFC 4271, in that it mandates what a BGP implementation
 > MUST do. See section 2 of the draft for the details. It's short and easy to read.
 > 
 > If we had noticed this earlier, we would have either chosen to home the document in IDR,
 > or explicitly made an exception to have GROW do the work. Given that we didn't, though,
 > the plan is to continue progressing the draft as a GROW document. However:
 > 
 > > Subject: Last Call: <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation
 > Behavior Without Policies) to Proposed Standard

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Keyur Patel <[email-masked]>
Date: Wed, 19 Apr 2017 22:16:14 +0000

And that would be good enough if that would allow exemptions of DC networks and any other networks that may need exemption.

In that case I support the publication.

Regards,
Keyur

On 4/19/17, 2:08 PM, "Jared Mauch" <[email-masked]> wrote:

    
    Jared Mauch

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 00:26:00 +0200

Keyur,

You can not set "insecure mode" before you reload the OS as current OS does
not have such knob. Unless you delay the deployment across N releases and
enforce sequenced upgrade.

The only way to prevent massive reachability failure upon reload due to
complete silent bgp prefix drop is to configure inbound policy for all EBGP
sessions before the reload and run with new image.

Of course this is all assuming that someone will read carefully the release
notes :)

If they do not the troubleshooting of this will be really painful ! CE will
see EBGP session as UP, will get all the routes and will send his routes.
CE will have no clue if PE dropped or accepted his routes. Likewise on the
other end .. Only imagine a network which has 10s of thousands of VPN CEs
as Bruno mentioned and their provider not following all releases CEs are
running.

At least doing it as part of OPEN msg will be immediately indicated to both
ends.

//R

On Thu, Apr 20, 2017 at 12:16 AM, Keyur Patel <[email-masked]> wrote:

> And that would be good enough if that would allow exemptions of DC
> networks and any other networks that may need exemption.
>
>     >>   Many IDR participants are also active in GROW and have had their
> say,
>     >> but if you haven't, now's your chance.
>     >>
>     >>   Thanks,
>     >>
>     >>   --John
>     >>
>     >>> Begin forwarded message:
>     >>>

---

From: Enke Chen <[email-masked]>
Date: Wed, 19 Apr 2017 15:53:46 -0700

Hi, Folks:

The document defines or changes the "default behavior" for EBGP.  However, the default
behavior for a particular code base or release was set long time ago, and in some cases
more than 20 years ago. To avoid breaking existing deployment in this case, the default
behavior in the code can not be changed (with or without this document). Then it becomes
a deployment practice for the policies to be configured.

So it seems to me that "Standard Track" may not be the right classification for this
document.  "Deployment recommendation or Practice" might be more appropriate.

Thanks.  -- Enke

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 01:13:32 +0200

Hi Enke,

100% agreed. I said the same to authors offline earlier today as well.

Every time you define a new AFI/SAFI one can make such AFI/SAFI mandatory
to have an inbound policy or not.

If authors would go that far and define new AFI/SAFI for IPv4 and IPv6
unicast so be it. The defaults there may be changed by such spec :) And
once accepted such new AFI/SAFI may share the routes with 1/1 & 2/1 for
IBGP propagation too.

Making it a Standards Track doc for all SAFIs MP-BGP is used today seems
like a pretty bad idea.

And as far as deployment practice we already have BCP document on this for
a while ... See section 6.3.1 of BCP194

REF: https://tools.ietf.org/html/bcp194#section-6.3.1

Cheers,
R.

On Thu, Apr 20, 2017 at 12:53 AM, Enke Chen <[email-masked]> wrote:

> > If we had noticed this earlier, we would have either chosen to home the
> document in IDR, or explicitly made an exception to have GROW do the work.
> Given that we didn't, though, the plan is to continue progressing the draft
> as a GROW document. However:
> >
> > - If anyone has a strong objection to the unusual procedure, please say
> so (either on-list, or to the chairs + AD).
> > - Please send any last call comments to the IETF LC (see below) although
> it's also OK to discuss here on the IDR list of course.
> >
> >> Subject: Last Call: <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP
> Route Propagation Behavior Without Policies) to Proposed Standard

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Wed, 19 Apr 2017 23:18:20 +0000

Hi Robert, Enke,

Also, irrespective of the Intended Status, the draft is conspicuously missing a “Backwards Compatibility” section. I would expect the draft to include this discussion even if it is progressed as BCP.

Thanks,
Acee

From: Idr <[email-masked]<mailto:[email-masked]>> on behalf of Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, April 19, 2017 at 7:13 PM
To: "Enke Chen (enkechen)" <[email-masked]<mailto:[email-masked]>>
Cc: Susan Hares <[email-masked]<mailto:[email-masked]>>, IDR List <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] IETF LC for IDR-ish document <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation Behavior Without Policies) to Proposed Standard

Hi Enke,

Every time you define a new AFI/SAFI one can make such AFI/SAFI mandatory to have an inbound policy or not.

If authors would go that far and define new AFI/SAFI for IPv4 and IPv6 unicast so be it. The defaults there may be changed by such spec :) And once accepted such new AFI/SAFI may share the routes with 1/1 & 2/1 for IBGP propagation too.

Making it a Standards Track doc for all SAFIs MP-BGP is used today seems like a pretty bad idea.

And as far as deployment practice we already have BCP document on this for a while ... See section 6.3.1 of BCP194

Cheers,
R.

On Thu, Apr 20, 2017 at 12:53 AM, Enke Chen <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Folks:

Thanks.  -- Enke

On 4/19/17 9:49 AM, John G. Scudder wrote:
>> From: The IESG <[email-masked]<mailto:[email-masked]>>
>> To: "IETF-Announce" <[email-masked]<mailto:[email-masked]>>
>> Cc: [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>
>> Reply-To: [email-masked]<mailto:[email-masked]>
>>
>>
>> [email-masked]<mailto:[email-masked]> mailing lists by 2017-05-02. Exceptionally, comments may be
>> sent to [email-masked]<mailto:[email-masked]> instead. In either case, please retain the
> [email-masked]<mailto:[email-masked]>

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: John Scudder <[email-masked]>
Date: Wed, 19 Apr 2017 23:25:08 +0000

(As an individual contributor)

On Apr 19, 2017, at 7:18 PM, Acee Lindem (acee) <[email-masked]> wrote:
> the draft is conspicuously missing a “Backwards Compatibility” section.

Seriously? "Backwards compatibility" in this case is "configure your router to do what it used to", right? We need a section to say that?

I must be missing something. 

Also, to Enke's "we can't ever change our defaults" point -- there are many RFCs. Some you probably comply with. Some you probably don't. How would this be different, assuming you elect not to change your implementation to comply?

Thanks,

--John

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 01:25:29 +0200

Well if this is a deployment BCP it does not need one ... you can apply any
policy on EBGP any time you like even today and nothing will break. At most
you upset or loose your customers.

But mandating to change behavior of an implementation default across all
SAFIs is a completely different animal.

On Thu, Apr 20, 2017 at 1:18 AM, Acee Lindem (acee) <[email-masked]> wrote:

> Hi Robert, Enke,
>
> Also, irrespective of the Intended Status, the draft is conspicuously
> missing a “Backwards Compatibility” section. I would expect the draft to
> include this discussion even if it is progressed as BCP.
>
> Thanks,
> Acee
>
> From: Idr <[email-masked]> on behalf of Robert Raszuk <
> [email-masked]>
> To: "Enke Chen (enkechen)" <[email-masked]>
> Cc: Susan Hares <[email-masked]>, IDR List <[email-masked]>
> Subject: Re: [Idr] IETF LC for IDR-ish document
> <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Wed, 19 Apr 2017 23:30:01 +0000

On 4/19/17, 7:25 PM, "John Scudder" <[email-masked]> wrote:

>Seriously? "Backwards compatibility" in this case is "configure your
>router to do what it used to", right? We need a section to say that?

Anytime one proposes to change the default behavior of a decades old
protocol to be more restrictive, I would expect this to be discussed.

Thanks,
Acee 

>Also, to Enke's "we can't ever change our defaults" point -- there are
>many RFCs. Some you probably comply with. Some you probably don't. How
>would this be different, assuming you elect not to change your
>implementation to comply?
>
>Thanks,
>
>--John

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 01:30:13 +0200

​John,

​> ​
How would this be different, assuming you elect not to change your
implementation to comply?

​Well if we are to standardize by rough consensus a RFC which we already
know is not going to be ​honored for the reasons clearly stated what are we
gaining ?

BGP implementations which support inbound policy to accept any routes will
continue doing so .. and those which do not also will continue not to do
so.

So what is the point ?

Thx,
r.

---

From: Enke Chen <[email-masked]>
Date: Wed, 19 Apr 2017 16:34:10 -0700

John,

I had "in this case" with the statement "the default can not be changed".
The reason is that the behavior change may completely cutoff connectivity
in this case.

Thanks.  -- Enke

---

From: Brian Dickson <[email-masked]>
Date: Wed, 19 Apr 2017 19:01:59 -0700

Hi, everyone,

I think the distinction is between "default" and "default", i.e. clearly
communicating what is meant by "default" in this document.

I think it would be perfectly fine for implementers to provide copious
"release notes" detailing what has changed, AND to apply whatever policy
gunk is necessary, on performing a software upgrade, to ensure no adverse
effects.

In other words, "default" here would mean the out-of-the-box behavior of an
otherwise unconfigured device, when BGP is enabled (regardless of AFI/SAFI
combinations).

By definition, a new deployment of a router can't cut off connectivity,
since it cannot come into existence with said connectivity.

Would making that distinction suffice, for everyone?

Brian

P.S. While I do applaud the vendor(s) for finally deciding not to break
things when code gets upgraded, separating upgrade from new deploy is all
that is required.

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 19 Apr 2017 19:56:30 -0700

+1 Enke!
Net every network engineer (unfortunately) follows IETF (or reads release notes), not every vendor clearly explains default behavior changes..

Regards,
Jeff

> On Apr 19, 2017, at 15:53, Enke Chen <[email-masked]> wrote:
> 
> Hi, Folks:
>

---

From: Enke Chen <[email-masked]>
Date: Wed, 19 Apr 2017 20:36:17 -0700

Hi, Brian:

I think that the backward compatibility concern is more about existing
deployment. For example, say an existing router does not have an inbound
policy and just accepts whatever routes from its provider, but it does 
have an outbound policy.  Let us further assume that the default behavior
in the software is to accept updates from a neighbor without an inbound
policy. 

Assume in the new code the default behavior is changed to drop updates from
a neighbor without an inbound policy.  Then as soon as the new software
is deployed on that router, the updates from the provider would be dropped
without any config changes.

Thanks.  -- Enke

On 4/19/17 7:01 PM, Brian Dickson wrote:
> Hi, everyone,
> 
> I think the distinction is between "default" and "default", i.e. clearly communicating what is meant by "default" in this document.
> 
> I think it would be perfectly fine for implementers to provide copious "release notes" detailing what has changed, AND to apply whatever policy gunk is necessary, on performing a software upgrade, to ensure no adverse effects.
> 
> In other words, "default" here would mean the out-of-the-box behavior of an otherwise unconfigured device, when BGP is enabled (regardless of AFI/SAFI combinations).
> 
> By definition, a new deployment of a router can't cut off connectivity, since it cannot come into existence with said connectivity.
> 
> P.S. While I do applaud the vendor(s) for finally deciding not to break things when code gets upgraded, separating upgrade from new deploy is all that is required.
> 
> On Wed, Apr 19, 2017 at 4:34 PM, Enke Chen <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     John,
> 
>     > On Apr 19, 2017, at 7:18 PM, Acee Lindem (acee) <[email-masked] <mailto:[email-masked]>> wrote:
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
> 
>

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 07:20:08 +0000

5) Isn't this problem statement/proposed solution a sub-part of the route leak issue?
IOW, wouldn't it be better addressed by/in draft-ymbk-idr-bgp-open-policy or draft-ietf-idr-route-leak-detection-mitigation ?

--Bruno

> From: [email-masked]  > Sent: Thursday, April 20, 2017 12:13 AM
> 
 > If so this would definitely not be deployment friendly. Especially for BGP/MPLS VPN
 > networks using EBGP for PE-CE routing and which have little use of filtering policies.
 > 
 > 4) Alternatively, there could be a (capability) signaling during the OPEN of the EBGP session.
 > With one end requesting this behavior to its peer. (or alternatively it's peer advertising the
 > presence of a policy and the receiver taking its own decision).
 > 
 > Possibly, some of the requirements may already be addressed by configuring a policy
 > limiting the number of routes acceptable from a peer/customers, and closing the EBGP
 > sessions when this limit is reached. This seems this would catch customers/peers advertising
 > the full routing by mistake/misconfiguration.
 > 
 > Thanks
 > Regards,
 > --Bruno
 > 
 >  > - As I understand it, the authors will add the Updates: 4271 header in addition to
 > potentially
 >  > taking in other comments from AD review.
 >  > - If anyone has a strong objection to the unusual procedure, please say so (either on-list,
 > or
 >  > to the chairs + AD).
 > ______________________________________________________________________
 > ___________________________________________________
 >

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 10:53:16 +0200

> mechanisms we are exploring where only knowledge of the mutual address

I know Cisco has been trying hard to market NX-OS as a data-center
centric device, not to be used on 'the Internet', however, folks do
connect NX-OS devices to the internet and this can result in pain for
others [1], especially when the defaults are insecure.

Kind regards,

Job

[1]: https://www.ietf.org/mail-archive/web/idr/current/msg16703.html

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 10:57:55 +0200

> sessions deployed with no policy?

Yes, you are missing the fact that a wild variety of defaults has been
deployed in the wild:

    o some reject on ingress, promiscuous on egress
    o some reject on both ingress and egress
    o some accept and announce everything

This documents aims to align those on a singular safe default: on EBGP
you'll need to configure a policy for something to happen.

> If so this would definitely not be deployment friendly. Especially for
> BGP/MPLS VPN networks using EBGP for PE-CE routing and which have
> little use of filtering policies.

This is what release notes are for.

> Is an expert policy tagging community eligible?

i do not understand what you mean. Can you elaborate or phrase
differently?

> Does this mean that this proposition should be restricted to Internet
> routing? (while BGP is used for many others applications)

It is restricted to EBGP.

> 4) Alternatively, there could be a (capability) signaling during the
> OPEN of the EBGP session. With one end requesting this behavior to its
> peer. (or alternatively it's peer advertising the presence of a policy
> and the receiver taking its own decision).
> 
> Possibly, some of the requirements may already be addressed by
> configuring a policy limiting the number of routes acceptable from a
> peer/customers, and closing the EBGP sessions when this limit is
> reached. This seems this would catch customers/peers advertising the
> full routing by mistake/misconfiguration.

no.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:03:10 +0200

> protocol to be more restrictive, I would expect this to be discussed.

This would be true if there were an actual existing default, however in
the wild we see that all kinds of implementation choices have been made.
It would also be helpful if we don't pretend that this hasn't been
discussed at length, for years.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:05:35 +0200

> continue doing so .. and those which do not also will continue not to do

Although I do not share your pessimistic view on what can and can't be
done, at the very least it will provide guidance for new BGP
implementations. 

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:09:41 +0200

> The reason is that the behavior change may completely cutoff connectivity

And very soon after the cutoff a customer may elect to roll back the
change and read the release notes!

The 'complete cutoff' only occurs after a specific sequence of events
which together would represent a cascading failure in due diligence. 

Given that we've suffered through decennia of insecure behaviour on some
platforms, I'd be careful to weigh the cost of such a self-inflicted
'complete cutoff' against the cost on internet operations in general to
persist in the folly behaviour that some platforms present.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:11:03 +0200

On Wed, Apr 19, 2017 at 07:56:30PM -0700, Jeff Tantsura wrote:
> +1 Enke!
> Net every network engineer (unfortunately) follows IETF (or reads
> release notes), not every vendor clearly explains default behavior
> changes..

Or is the reverse true, that IETF participants have not been tracking
operational issues? 

"People may not read release notes" is an extremely weak justification
for insecure behaviour.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:16:10 +0200

On Thu, Apr 20, 2017 at 07:20:08AM +0000, [email-masked] wrote:
> 5) Isn't this problem statement/proposed solution a sub-part of the
> route leak issue?  IOW, wouldn't it be better addressed by/in
> draft-ymbk-idr-bgp-open-policy or draft-ietf-idr-route-leak-detection-mitigation ?

Why do you think so?

Kind regards,

Job

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:28:59 +0000

Job,

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 10:58 AM
> 

My comment still applies to some implementations and some deployments.
 
 > This is what release notes are for.

Possible awareness does not change the point that this forced change of default behavior is not deployment friendly for non Internet applications.
 
 > i do not understand what you mean. Can you elaborate or phrase
 
Can this document define what he means by "configured export policy".  Possibly by referring to standardized yang models.
Alternatively, it looks like the document means that "by default a BGP speaker MUST NOT advertise any routes to an EBGP peer". Which would solve the problem by not referring anymore to configured export policy.
 
 > It is restricted to EBGP.

Yes, also, but this seems orthogonal to me.
So to rephrase, does this mean that this proposition should be restricted to EBGP sessions used for Internet routing? (as EBGP is used for many others applications)

 > > full routing by mistake/misconfiguration.
 
Can this document better describe the problem statement?
Please find below my reading of the current text

"   There are BGP routing security issues that need to be addressed to make the Internet more stable."
Agreed, but not very specific.

"  Route leaks [RFC7908] are part of the  problem, but software defects or operator misconfigurations can  contribute too."
- Route leaks seem to be already worked on by draft-ymbk-idr-bgp-open-policy or draft-ietf-idr-route-leak-detection-mitigation. Plus this document is not a solution to route leak
- Software defect won't probably be addressed by this proposal
- Misconfiguration won't probably be addressed by this proposal.

So, although I agree that requiring explicit export and import for EBGP session used in the Internet is probably an improvement, I disagree with changing the default of existing deployed EBGP sessions for non Internet usages, as this would breaks current deployment.
At minimum, this should be restricted to new configured sessions ... but even this would break existing provisioning scripts so this is still not so deployment friendly.

King regards,
-- Bruno

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:30:34 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 11:16 AM
> 
 
Route leak is mentioned in the introduction (read justification) of this document.

Kind regards,
--Bruno

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:32:44 +0000

> From: Idr [mailto:[email-masked]] On Behalf Of Job Snijders  > Sent: Thursday, April 20, 2017 11:06 AM
> 
 > > ​> 

This would work for me.
Can the document be updated to reflect this?

Thanks,
Kind regards,
--Bruno

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:35:13 +0200

> Route leak is mentioned in the introduction (read justification) of

A mention of the phrase "route leak" does not constitute as to why
draft-ymbk-idr-bgp-open-policy or
draft-ietf-idr-route-leak-detection-mitigation are better at addressing
the problem of insecure defaults. Perhaps the drafts complement each
other?

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:37:06 +0200

> Can the document be updated to reflect this?

Existing deployments are just that: existing deployments. It seems
superfluous to mention that compliance with the Internet-Draft might be
enforced only after a software upgrade. The same goes for many other
RFCs.

Kind regards,

Job

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:40:50 +0000

> From: Idr [mailto:[email-masked]] On Behalf Of Job Snijders  > Sent: Thursday, April 20, 2017 11:10 AM
> 

The document is asking network operators which have correctly configured their network to retrofit the configuration of 1000s of EBGP configurations, in order to accommodate some operators which are not capable of correctly configuring their EBGP session.
Plus the solution is asking those later operators to do a configuration, while the assumption is that they can't be trusted to correctly configured their EBGP session. Why do you think that they won't just copy/past 1 additional line of configuration in order automatically distribute all the routes for each new EBGP session?

Kind regards,
--Bruno

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:50:13 +0000

> From: Idr [email-masked]  > Sent: Thursday, April 20, 2017 11:29 AM
> 
 > Job,
 > 
 >  > > I admit that I was not following this subject so my comments might be redundant or
 > even
 >  > stupid.
 >  > >
 > Possible awareness does not change the point that this forced change of default behavior is
 > not deployment friendly for non Internet applications.
 > 

Because I think the document means "configured route (re)distribution/advertisement policy".
Possibly this was implied by the word "export" but one could read this as any outbound policy, while a policy setting some communities would probably not help much.

--Bruno (sorry for the spam)

 > Alternatively, it looks like the document means that "by default a BGP speaker MUST NOT
 > advertise any routes to an EBGP peer". Which would solve the problem by not referring
 > anymore to configured export policy.
 > 
 >  > > 3) From the introduction
 > So to rephrase, does this mean that this proposition should be restricted to EBGP sessions
 > used for Internet routing? (as EBGP is used for many others applications)
 > 
 >  >
 > - Route leaks seem to be already worked on by draft-ymbk-idr-bgp-open-policy or draft-ietf-
 > idr-route-leak-detection-mitigation. Plus this document is not a solution to route leak
 > So, although I agree that requiring explicit export and import for EBGP session used in the
 > Internet is probably an improvement, I disagree with changing the default of existing
 > deployed EBGP sessions for non Internet usages, as this would breaks current deployment.
 > At minimum, this should be restricted to new configured sessions ... but even this would
 > break existing provisioning scripts so this is still not so deployment friendly.
 > 
 > King regards,
 > -- Bruno
 > 
 >  > Kind regards,
 >  >
 >  > Job
 >

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:50:31 +0200

On Thu, Apr 20, 2017 at 09:40:50AM +0000, [email-masked] wrote:
> The document is asking network operators which have correctly
> configured their network to retrofit the configuration of 1000s of
> EBGP configurations, in order to accommodate some operators which are
> not capable of correctly configuring their EBGP session.

No, the beneficiaries of this draft are the users of Internet at large,
not just 'incapable operators'.

It is ludicrous to argue that an operator has 1000s of EBGP
configurations but would not be able to accomodate any changes in the
software's behaviour. A parallel can be drawn with the 'IOS
small-servers' which over time were disabled by default, rather then
enabled. I don't recall any outcry over that either.

> Plus the solution is asking those later operators to do a
> configuration, while the assumption is that they can't be trusted to
> correctly configured their EBGP session. Why do you think that they
> won't just copy/past 1 additional line of configuration in order
> automatically distribute all the routes for each new EBGP session?

Ah, there may lay our misunderstaanding. We do not blame operators or
make any judgement on whether they can be trusted or not. 

The problem is that there are some platforms which _immediately_
activate a line of configuration after you press enter, and a BGP
session may already come up before you get to the configuration line
which restricts what should be announced or accepted. This draft
rectifies that behaviour to 'start closed' (i guess this is a variant of
the "fail closed" concept) rather then 'start open', or differently
phrase: one should not start insecure.

Kind regards,

Job

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:52:46 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 11:35 AM
> 

Eventually.
I'm asking for this discussion to happen.

Kind regards,
--Bruno

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 11:56:24 +0200

On Thu, Apr 20, 2017 at 09:50:13AM +0000, [email-masked] wrote:
>  > Can this document define what he means by "configured export
>  > policy".  Possibly by referring to standardized yang models.
> 
> Because I think the document means "configured route
> (re)distribution/advertisement policy".  Possibly this was implied by
> the word "export" but one could read this as any outbound policy,
> while a policy setting some communities would probably not help much.

We purposefully did not specify what the contents of a policy
constitute, or whether policies should have an implicit 'accept'
terminal clause or an implicit 'deny' terminal clause.

All we try to accomplish, is that if you configure _nothing_ for an EBGP
peer, the software will not advertise any routes or use any routes which
it receives. 

Kind regards,

Job

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 09:58:25 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 11:37 AM
> 

The question was whether this new default policy could be proposed (or imposed) as a BCP only for _new_ implementations.
And by new implementations, I mean new source code (e.g. RtBrick) not a software upgrade of an existing code.

Kind regards,
--Bruno

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 10:13:12 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 11:51 AM
> 
 
My text was not discussing about beneficiaries, so I don't think that "No" applies.
 
 > software's behaviour.

That's your opinion, but you can't speak for others.
e.g. managing a few high bandwidth Internet connection, on a very few locations, is very different than managing 1000s of customers CE in 1000s of locations. Especially as breaking the connectivity would requiring sending a tech person on customer's site.

> A parallel can be drawn with the 'IOS
 
It's not the same thing to disable a feature that is not used, than disabling a feature that is used and critical for the connectivity hence the business.
 
 > make any judgement on whether they can be trusted or not.

The point is not about blaming. The point if that if you don't trust X, you don't ask X to perform the check/policy insurance.
 
 > activate a line of configuration after you press enter

Here we comes.
If that is the problem, then may be this is the problem which needs to be solved.
Nothing related to BGP or even the IETF.

>, and a BGP
 
If this is the problem, may be the CLI should first mandate for the BGP policy to be applied, and then allow the configuration of the peer (e.g. its IP address)
Again, not a protocol or IETF issue. (although netconf may help as I think it provide support for transaction)

Thanks for engaging the discussion and the useful chat.

Kind regards,
Bruno

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 12:11:18 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Thursday, April 20, 2017 11:56 AM
> 

People do not establish EBGP session for the pleasure of the OPEN message. They enable it to get and receive routes. So without any additional thinking, they will type whatever additional keyword that you may require.

e.g. 
OLD:  neighbor 10.1.1.1 remote-as 65124 
NEW: neighbor 10.1.1.1 remote-as 65124 send-routes receive-routes
Or NEW2 
neighbor 10.1.1.1 remote-as 65124
neighbor 10.1.1.1 send-routes receive-routes

People will just blindly use the new CLI. So this would only address the situation where the CLI interprets the command on a line per line basis and send all routes without waiting for the policy/whole BGP session configuration to be configured. (e.g. waiting for the ending delimiter).
If so, this looks like a CLI/vendor specific issue. Not a BGP / IETF one.
And definitely, we don't need to impact/change the behavior for the BGP sessions which are already configured.

Kind regards,
--Bruno

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:33:44 -0400

> On Apr 19, 2017, at 6:16 PM, Keyur Patel <[email-masked]> wrote:
> 

I’m not sure what makes DC networks unique to be exempt from adding a one line policy to their configuration, this is a very low bar similar to configuring a hostname or AAA authentication that a DC network would have come from their ZTP/automation.

I find this incredibly worrisome thinking, next we will have an exemption for government run networks to what end?

- Jared

---

From: "i3D.net - Martijn Schmidt" <[email-masked]>
Date: Thu, 20 Apr 2017 15:37:28 +0200

> I find this incredibly worrisome thinking, next we will have an exemption for government run networks to what end?
Hi all,

DC network operator here! As long as it's well documented, adding a new 
line to my configuration shouldn't be a problem when I'm doing an 
upgrade during a maintenance window anyway. Moreover, we operate our 
network on a "trust nothing" basis even for our internal BGP sessions.

Best regards,
Martijn Schmidt
i3D.net / AS49544

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:40:13 -0400

> On Apr 19, 2017, at 6:26 PM, Robert Raszuk <[email-masked]> wrote:
> 
> Keyur,
> 
> You can not set "insecure mode" before you reload the OS as current OS does not have such knob. Unless you delay the deployment across N releases and enforce sequenced upgrade.

Infact, this is the recommendation that I’ve provided to vendors that have expressed concerns.  There are many defaults that have not always been displayed, but things like IOS have “show run all” so you can see these.  

Something like the ‘bgp unsafe-ebgp-policy’ could be generated on their respective implementations.  I didn’t think that GROW/IDR needed to tell implementors this level of how to manage their release, so this does seem somewhat out of scope, but a concern I can see needs to be thought about.

> The only way to prevent massive reachability failure upon reload due to complete silent bgp prefix drop is to configure inbound policy for all EBGP sessions before the reload and run with new image. 

Since we’re talking about how to operate a network:
- People who are taking advantage of an undefined behavior will always be surprised
- Vendors can take the N+1.x and N+2.x release strategy, where in N and N+1 they generate their equivalent of IOS-XR and the "bgp unsafe-ebgp-policy” policy to prevent their customers from breaking
- In a release N+2(or more) that would become the “default”.

> Of course this is all assuming that someone will read carefully the release notes :) 

Most people don’t, and I’ve always suggested to vendors they implement some sort of incremental approach to resolving this.

What worries me is that there is a major incumbent provider who doesn’t see this as the serious (and well-documented) security issue that it is for those operating large networks.

> If they do not the troubleshooting of this will be really painful ! CE will see EBGP session as UP, will get all the routes and will send his routes. CE will have no clue if PE dropped or accepted his routes. Likewise on the other end .. Only imagine a network which has 10s of thousands of VPN CEs as Bruno mentioned and their provider not following all releases CEs are running. 

If they don’t know how to troubleshoot BGP, that isn’t the vendors fault.

> At least doing it as part of OPEN msg will be immediately indicated to both ends. 

This is you promoting a different draft, I recommend another thread for that draft.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:41:02 -0400

> On Apr 19, 2017, at 6:53 PM, Enke Chen <[email-masked]> wrote:
> 
> Hi, Folks:
> 

Please see my other thread on this topic.

I’m disappointed to see Cisco coming out for BGP insecurity.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:43:11 -0400

> Also, irrespective of the Intended Status, the draft is conspicuously missing a “Backwards Compatibility” section. I would expect the draft to include this discussion even if it is progressed as BC…

Are we to dictate to the vendors how to write a security-conscious BGP implementation?  I’m seeing one vendor that has it more-than-one way as default saying uniformity is impossible.

Once again, similar to other discussions that transpired in 2016 I’m concerned that people in IDR are disconnected from the operational practices of the internet-at-large.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:49:01 -0400

> On Apr 19, 2017, at 11:36 PM, Enke Chen <[email-masked]> wrote:
> 
> Hi, Brian:
> 

Vendors have changed defaults before and incrementally done so, take for example
a well known vendor that had problems that arose during an IETF meeting regarding
configuration options like “service tcp-small-servers” and “service udp-small-servers”
which eventually became a) exposed and b) non-default.

Surely there is precedent for a vendor to identify a release strategy that resolves
the operational concerns without an IETF document and IDR dictating to the vendor how
exactly to perform their release cycle.  I think we all know that’s infeasible and is
entirely a straw man argument for not addressing the well documented operational insecurity
introduced by clinging to an exploitable practice.

Let me reiterate, it’s not even like all vendors are internally consistent across their
releases here, this is codifying a safe default, such as not running an open relay for
delivery of SPAM or “ip directed-broadcast” for things like the SMURF attacks.

- Jared

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 15:50:55 +0200

Jared,

- Vendors can take the N+1.x and N+2.x release strategy, where in N and N+1
> they generate their equivalent of IOS-XR and the "bgp unsafe-ebgp-policy”
> policy to prevent their customers from breaking

​In the past things like that were also actually solved within single
release by automatically generating this line of "missing" configs if no
other policy was configured. However for this specific case I am not sure
what does it buy you in practice. Maybe consistency across BGP
implementations.  ​

> This is you promoting a different draft, I recommend another thread for

​Well if both drafts can solve the same problem maybe there is a room to
discuss it and pick one to go forward with.

//R
​

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:51:27 -0400

> On Apr 20, 2017, at 5:40 AM, <[email-masked]> <[email-masked]> wrote:
> 

My goal isn’t perfection, but improvement.  There will always be people who do nonsensical things, this is an attempt to address those doing it unintentionally :-).

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:55:39 -0400

> On Apr 20, 2017, at 5:50 AM, <[email-masked]> <[email-masked]> wrote:
> 

IMO:

someone writing the policy statement saying 

term { accept; } 

is enough to satisfy my concerns, it’s when people do nothing and are left sending their full rib back to their upstream causing load it’s a problem.  While we’ve moved past the days of 11.1&12.0 in IOS land where there was a high cost on the CPUs to x86 compute where the burden is less, it’s still additional processing and a resource consumer.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 09:58:30 -0400

> On Apr 20, 2017, at 5:58 AM, <[email-masked]> <[email-masked]> wrote:
> 

My existing $dayjob vendors should expect we would not purchase equipment that is not secure out of the box.  If product management can’t get behind a strategy of secure devices, this speaks volumes to me and will impact our purchasing.

- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 10:06:26 -0400

> On Apr 20, 2017, at 9:50 AM, Robert Raszuk <[email-masked]> wrote:
> 
> Jared,
> 
> ​In the past things like that were also actually solved within single release by automatically generating this line of "missing" configs if no other policy was configured. However for this specific case I am not sure what does it buy you in practice. Maybe consistency across BGP implementations.  ​

I see value in consistent behaviors.  We have vendors that do different things here, and are inconsistent amongst themselves as well.  I’m surprised that PM types haven’t pushed for a consistent behavior, but this may more reflect internal company cultures.

> ​Well if both drafts can solve the same problem maybe there is a room to discuss it and pick one to go forward with. 

I think the other drafts have some value, but I’ve not yet been able to wrap my head around where the technical and business pieces intersect and would cause operational issues.  I’ll leave my detailed comments for the other document, but they have been raised at the microphone at the past 2 WG meetings re: the Open message draft.  (let me go find the thread or start some comments on those).

- Jared

---

From: "Dickinson, Ian" <[email-masked]>
Date: Thu, 20 Apr 2017 14:50:36 +0000

Inline with [IanD]

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jared Mauch
Sent: 20 April 2017 14:56
To: [email-masked]
Cc: [email-masked]; Hares Susan <[email-masked]>
Subject: Re: [Idr] IETF LC for IDR-ish document <draft-ietf-grow-bgp-reject-05.txt> (Default EBGP Route Propagation Behavior Without Policies) to Proposed Standard

IMO:

term { accept; }

[IanD] The larger issue here is not local load causing impact to the "perpetrator" but the leak effectively hijacking traffic that has a much larger blast radius. I've been annoyed at implementations failing open for years, and I'd like to see this addressed. I think this draft at least codifies good behaviour and I support it.

- Jared

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 08:36:17 -0700

Hi, Jared:

If it is not obvious, let me state the I participate in IDR WG as an individual
contributor, just like you do I suppose.

Let me rephrase what I said, for a code base with a large and diverse customer base
I do not foresee any possibility for the default behavior change in this case. Again
please treat it as my personal opinion.

I am certainly aware of other cases where the default behavior has been changed. But
this one is different.  This case seems similar to the default behavior ("permit" or
"deny") for an empty ACL.  Once the "permit" or "deny" is set in the code and is
widely deployed for a long time, it is just not possible to make the switch between
"permit" and "deny".

Regards,  -- Enke

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 17:41:42 +0200

On Thu, Apr 20, 2017 at 08:36:17AM -0700, Enke Chen wrote:
> If it is not obvious, let me state the I participate in IDR WG as an
> individual contributor, just like you do I suppose.
> 
> Let me rephrase what I said, for a code base with a large and diverse
> customer base I do not foresee any possibility for the default
> behavior change in this case. Again please treat it as my personal
> opinion.
> 
> I am certainly aware of other cases where the default behavior has
> been changed. But this one is different.  This case seems similar to
> the default behavior ("permit" or "deny") for an empty ACL.  Once the
> "permit" or "deny" is set in the code and is widely deployed for a
> long time, it is just not possible to make the switch between "permit"
> and "deny".

You say "it is not possible", however there are examples where it turned
out to be possible. At least one implementer confirmed on this list that
they plan to change their default.

Perhaps, "impossible" should be phrased as "unwilling".

Kind regards,

Job

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 08:57:07 -0700

Job,

It depends on the customer base and also how long the software has been deployed.
Just think about the scenario that a large number of customers would lose network
connectivity unexpectedly due to a default behavior change in the code. Such outages
could keep happening to different customers for years to come.

Perhaps, changing "impossible" to "impractical" :-)

Regards,  -- Enke

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 11:58:58 -0400

> contributor, just like you do I suppose.

	Of course.

	:-)

> widely deployed for a long time, it is just not possible to make the switch between

	I think there is a very distinct issue here, I would of course love
for existing vendors to come in compliance with a "secure-by-default" strategy.
Many othr people with far more devices than BGP speakers have seen this
necessity and made the change, claiming that BGP is somehow unique or that
operators do not deserve this safety is inconsiderate.

	My intent isn't to update 4271, it's Alvaro that has raised that
spectre.

	Cisco (as an example) has many cases where defaults have changed,
the same is true of many pieces of software, either in development or after
deployment.  Sometimes thse are called bugs, sometimes a feature.

	I've outlind my recommendation of how I would like to see 
nvgen function work in Ciscos case, that these devices can simply
expose these defaults, just as config bits often are massively reordered
betwen even minor interims or rebuilds.

	If you take the line-by-line interpreted vs atomic model of
driving configuration, it is not possible to apply policy to a session before
the FSM starts running.  This is unsafe.

	I'm seeing the GROW WG advise what an operational safe default
practice is as there is no codified standard of the behavior.  Operators
value consistent behavior, and it's clear with much of the ongoing work
in IETF around standards based models that the vendors see value as well.

	I suspect many people in IDR have not seen a bank be MITM'ed
for months due to routing leakage.  They would say this must be the
intended policy of 'pass all'.

	With further BGP speakers coming online, we need to address
this technical debt that many of us bear the burden of.

	Re: the draft under discussion, did you look at any of the prior
versions of the text?  I suspect that many people have not.

The pre-GROW-WG text said:

https://tools.ietf.org/html/draft-mauch-bgp-reject-00

-- snip --
3.  Solution Requirements

   The following requirements apply to the solution described in this
   document:

   o  Software MUST NOT accept routes from an eBGP peer without an
      operator configuring a policy

   o  Software MUST NOT require a configuration directive to operate in
      this mode.

   o  Software MUST NOT send routes to an eBGP peer without an operator
      configuring a policy

   o  Software MUST provide protection from internal failures preventing
      the advertisement and acceptance of routes

   o  Software MAY provide a configuration option to disable this
      security capability.
-- snip --

After advisement of BGP implementors, and post-adoption it seemed
an implementor may require a pointer to actually when and how to
reject the routes, as this guidance was seen as unclear.

This is when section 3 was updated:

https://tools.ietf.org/html/draft-ietf-grow-bgp-reject-00

-- snip --
   o  Software MUST mark any routes from an eBGP peer as 'invalid' in
      the Adj-RIB-In, if no explicit policy was configured.
-- snip --

And eventually morphed into something that cited a Informative Reference
and gave detailed guidance:

https://tools.ietf.org/html/draft-ietf-grow-bgp-reject-05

-- snip --

   o  A BGP speaker MUST consider any routes advertised by an EBGP peer
      ineligible for route selection (section 9.1.1 [RFC4271]), if no
      import policy was configured for the peer.

-- snip --

Is one of these previous iterations of the text more acceptable to you
or the IDR WG?

	- Jared

-- 
Jared Mauch  | pgp key available via finger from [email-masked]
clue++;      | http://puck.nether.net/~jared/  My statements are only mine.

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 12:07:36 -0400

> Perhaps, changing "impossible" to "impractical" :-)

	I'd like to call it well-considered. :-)

	I'm operating a network with Juniper, NX-OS, IOS-XR, IOS-Classic, IOS-XE,
and various implementations that require custom policy to be implemented.

	There can be a path forward plotted that would prevent currently
deployed people from having issues, we're surely bright enough to do that.

	To make it clear: I don't want to break someones routers.

	I do want to make it harder for someone to leak a table when they
have a new router.

	I don't belive the bar should be high, it can be embedded in whatever
configuration/ZTP/automation/cut+paste template out there.  It could come
in the form of yang over netconf, or a DHCPv6/DHCPv4 option.  It could
come from a TXT record in DNS, or wahtever configuration method the vendor
invents that is new and unimagined by th WG today.

	I don't feel it requires updating 4271 to attain that goal, it's
clear implementors have seen a path to do this today without having
a concern with 4271, and I believe that Alvaro is wrong in the presumption
this document updates 4271.  (I'm also willing to be told that I'm too rough
for consensus :-).

	- Jared

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 18:43:14 +0200

On Thu, Apr 20, 2017 at 08:57:07AM -0700, Enke Chen wrote:
> It depends on the customer base and also how long the software has
> been deployed. Just think about the scenario that a large number of
> customers would lose network connectivity unexpectedly due to a
> default behavior change in the code. Such outages could keep happening
> to different customers for years to come.

If these outages occur, they'll be quickly remedied since the service is
down, which provides incentive to either roll back or deploy a fix.  We
call this "fail hard". However, it is a failure mode that is
preventable, and vendors have a big role in this.

This outage only occurs if and only if there a sequence of process
errors that together are a cascading failure: e.g. absolutely no reading
or reviewing of the release notes by anyone in the organisation, no
taking heed of prior notifications (through for instance operational
mailing lists or customer/vendor meetings), no testing, and no staggered
canary deployment, all of this on top of reliance on the operating
system's default (whatever it may be) when there is no policy configured
for the EBGP peer. 

Why would anyone download a new software if one are not going to read
the release notes? Why would one upgrade a software if one does not know
what it will do? Any problems will be self-inflicted, and easy to remedy
by rolling back or configuring a policy.

Your perceived risk, which can be managed, does not legitimize a
continuation of inconsistent or insecure behaviour.

Kind regards,

Job

---

From: <[email-masked]>
Date: Thu, 20 Apr 2017 16:47:14 +0000

Jared,

> From: Jared Mauch > Sent: Thursday, April 20, 2017 6:08 PM
> 

Have you considered starting solving this issue with yang model? Although this is more long term, this should be the target/long term solution and hence this has a significant value. Also the installed based is smaller so change would be easier. And finally, this probably natively only applies to new EBGP sessions, i.e. would not affect existing deployed based.
e.g. the (mandatory) default import/export policy would be to filter all routes. 
 
Regards,
--Bruno (just trying to propose something, feel free to ignore)

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 09:50:53 -0700

Job,

My personal opinion:

Vendors are *not* in the business of intentionally creating network outages :-)
Those that do may not stay in the business for long :-)

Regards,  -- Enke

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 18:57:57 +0200

> Those that do may not stay in the business for long :-)

You could also argue, that by not implementing the bgp-reject guidance,
a vendor intentionally creates outages which result from inadvertent
propagation of full routing tables.

If a vendor implements bgp-reject, the customers enjoy mitigation of the
risk of hitting the timer penalty on a 'maximum prefix reached'
shutdown, or perhaps someone configured their router to not
automatically re-enable the session after maximum-prefix is hit, in this
case the outage prolongs.

If you are committed to preventing outages, implementing bgp-reject
seems the right thing to do.

Kind regards,

Job

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 10:08:50 -0700

Job,

I have nothing new to add :-(

Regards,  -- Enke

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 19:15:07 +0200

Job,

Enke observed a difference between operational outages and vendor
implementation caused outages. Those are different things so I suggest to
separate them.

But as I pointed out two ways already it is doable to implement it in a way
that outage will not happen even if someone would never read release notes
and upgrade routers via web gui with new OS.

So instead this ping pong how about we focus on point what constitutes a
policy and what you expect associated behavior of complaint implementation
to be.  I asked that already but your answer that you left it loosely
defined does not seems good enough for the spec implementer.

*A* Must it be per neighbor ? Can it be per VRF/table/bRIB  ?

*B* Does enabling BGP Origin Validation should be considered as such policy
or not ?

*C* Should received routes still reside in Adj_RIB_In ? Should BMP still
send all routes to collector even if those would be considered not
complaint with RFC ?

*D* If we do eBGP auto discovery in IX env of peers would that overwrite
such RFC ?

*E* If someone does LLDP based eBGP peer discovery on p2p would that also
be exempt from such enforcement ?

Thx a lot,
Robert.

---

From: Gert Doering <[email-masked]>
Date: Thu, 20 Apr 2017 19:29:04 +0200

Hi,

> *A* Must it be per neighbor ? Can it be per VRF/table/bRIB  ?

per neighbour, or per peer-group/neighbour-group

> *B* Does enabling BGP Origin Validation should be considered as such policy

certainly not for egress.

for ingress, it *is* a policy, so I'd see it as "good enough"

> send all routes to collector even if those would be considered not

I see this as an issue of minor importance.  Both approaches serve the
goal of avoiding to send or accept prefixes unless said activity is
enabled by policy.

> be exempt from such enforcement ?

Both would be something that the box wouldn't do out of the box, so 
I would expect some sort of parent/inheritance config for IX/p2p 
magic config, which would be able to specify policy.

As such, *of course* eBGP sessions at IXPs or p2p to customers/peers/
upstreams, MUST follow the same rules: no policy = no prefixes.

Which is the whole point: do not leak full tables to any random peer, 
unless you *want* to do so - and if so, it must be turned ON.

Gert Doering
        -- Operator, having received his share of full-table leaks over time
-- 
have you enabled IPv6 on something today...?

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 17:36:41 +0000

> per neighbour, or per peer-group/neighbour-group

The purpose is per neighbor. Questioning this aspect makes me suspect you
might not grok the concept and problem space at all.

> for ingress, it *is* a policy, so I'd see it as "good enough"

BGP origin validation in itself is not a policy, but a database against
which you can lodge lookups. An operator has to configure the device to
drop or accept RPKI invalids. If the operator has formulated that policy
and associated it with the session, you have a policy.

> > send all routes to collector even if those would be considered not

Alvaro suggested text to that extend.

> *D* If we do eBGP auto discovery in IX env of peers would that overwrite

Why would it?

> > be exempt from such enforcement ?

Why would it?

Kind regards,

Job

---

From: "Alvaro Retana (aretana)" <[email-masked]>
Date: Thu, 20 Apr 2017 17:50:36 +0000

Jared:

Hi!

Not everyone in this thread was part of the initial conversations we had, so to give a little background:  I think (yes, still) that the document (as is in -05) should be marked as updating rfc4271 because it starts off saying that it “defines the default behavior of a BGP speaker…” and later (Section 2) describes specific changes pointing at pieces of rfc4271: “…MUST consider any routes advertised by an EBGP peer ineligible for route selection (section 9.1.1 [RFC4271])…”.

Again, what gives me heartburn and the reason this document caught my attention is that change in the default – and from this thread, I can see that is an issue for others.

Reading what you wrote below, about other potential options to achieve the same result, I agree with you that the bar doesn’t have to be as high as changing the default in rfc4271.  But the current document doesn’t reflect that.

Maybe what we need is to describe the solution in a way that is not so rfc4271-specific.  Explain what the behavior should be (not how to achieve it), and even talk about the operational pain, and what operators should consider with the current not-specified behavior (which the document doesn’t do much of now).  I think that would be a very different document with a different set of discussion points, but one that could lead to the goal.

During the early thread of this draft (a couple of years ago), several people suggested that the status should be a BCP.  I can see how a document explaining the pains and the considerations for Internet routers could be a BCP.

Just trying to move the conversation forward.

Alvaro.

On 4/20/17, 12:07 PM, "Jared Mauch" <[email-masked]> wrote:

---

From: Brian Dickson <[email-masked]>
Date: Thu, 20 Apr 2017 11:08:56 -0700

TL;DR: BCPs don't work; operators vary in experience, diligence, etc;
vendors don't have common defaults; "open", "route-leaks", and "reject"
work differently and are all needed; code changes will have a long tail of
non-upgraded routers for a long time.

BCPs don't work - see BCP38 for the canonical example.

If operators were consistent (as in globally, uniform, ubiquitous without
exception) in applying best practices, these proposals would not even
exist. Their mere existence is proof that they are needed.

"reject" will only help once implemented, on upgraded or newly deployed
routers; ditto for "open"; "route-leaks" requires operator configuration,
but uses a transitive-optional attribute (and thus works alongside
non-upgraded routers). All three are needed, for their respective
attributes (on by default; stop leak origination; limit leak propagation).

The long tail on upgrades is what makes all three needed. No single one is
sufficient by itself, unless/until every router has been upgraded AND
configured. The three together are an imperfect but scalable and
incrementally deployable set of improvements with real global results.

Trying to move things in a productive direction.

My $0.02.

Brian

On Thu, Apr 20, 2017 at 10:50 AM, Alvaro Retana (aretana) <[email-masked]
> wrote:

> Jared:
>
> Hi!
>
> Not everyone in this thread was part of the initial conversations we had,
> so to give a little background:  I think (yes, still) that the document (as
> is in -05) should be marked as updating rfc4271 because it starts off
> saying that it “defines the default behavior of a BGP speaker…” and later
> (Section 2) describes specific changes pointing at pieces of rfc4271:
> “…MUST consider any routes advertised by an EBGP peer ineligible for route
> selection (section 9.1.1 [RFC4271])…”.
>
> Again, what gives me heartburn and the reason this document caught my
> attention is that change in the default – and from this thread, I can see
> that is an issue for others.
>
>
> Reading what you wrote below, about other potential options to achieve the
> same result, I agree with you that the bar doesn’t have to be as high as
> changing the default in rfc4271.  But the current document doesn’t reflect
> that.
>
> Maybe what we need is to describe the solution in a way that is not so
> rfc4271-specific.  Explain what the behavior should be (not how to achieve
> it), and even talk about the operational pain, and what operators should
> consider with the current not-specified behavior (which the document
> doesn’t do much of now).  I think that would be a very different document
> with a different set of discussion points, but one that could lead to the
> goal.
>
> During the early thread of this draft (a couple of years ago), several
> people suggested that the status should be a BCP.  I can see how a document
> explaining the pains and the considerations for Internet routers could be a
> BCP.
>

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 20 Apr 2017 11:24:24 -0700

Can't miss that food fight ;-) So couple observations:

i) as often, when an RFC tries to mandate anything beyond things on the
wire & the procedures to treat it, it tries ultimately to mandate
implementations and this is seldom wise
ii) +1 Enke, +1 Acee.  I understand operators here wanting "their default"
but large, diversified codebases have to serve (especially with BGP) a wide
spectrum of customers with different needs and with that, different
defaults make sense. And I remember routers booting up with a question list
of "press 1 if you're an evil syndicate, press 2 if you're a bedroom ISP,
press 3 if you think this is a toaster" to get the right defaults set ...

Having said that, I think this is BCP material at best and if this is a BCP
then

i) a "backward compatibility a.k.a which end of the stick is sharp" section
is very advisable
ii) the BCP should describe which customer segment is best served with
which default

--- tony

On Thu, Apr 20, 2017 at 11:08 AM, Brian Dickson <
[email-masked]> wrote:

> On Thu, Apr 20, 2017 at 10:50 AM, Alvaro Retana (aretana) <
> [email-masked]> wrote:
>
>> Jared:
>>
>> Hi!
>>
>> Reading what you wrote below, about other potential options to achieve
>> the same result, I agree with you that the bar doesn’t have to be as high
>> as changing the default in rfc4271.  But the current document doesn’t
>> reflect that.
>>

-- 
*We’ve heard that a million monkeys at a million keyboards could produce
the complete works of Shakespeare; now, thanks to the Internet, we know
that is not true.*
—Robert Wilensky

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 20 Apr 2017 14:31:29 -0400

I agree that the document represents an update to RFC 4271, for the reasons you give.

I don't see any advantage to be gained by going back to square one as you talk about in your later paragraphs. If the goal of the authors is to say what what a BGP implementation MUST do (and as far as I can tell, it is) then it seems to me that this is the right vehicle. Furthermore, as of now I don't see any reason to despair of coming to consensus. Probably we should let the LC play out and then see where we've landed, don't you think? Maybe we will have (rough) consensus to advance the document; in that case, done. Maybe we will not; in that case we have to decide what to do next.

--John

> On Apr 20, 2017, at 1:50 PM, Alvaro Retana (aretana) <[email-masked]> wrote:
> 
> Jared:
> 
> Hi!
> 
...

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 20 Apr 2017 14:40:52 -0400

Enke,

> Perhaps, changing "impossible" to "impractical" :-)

Various people have provided worked examples of how they think this change could be effected without causing the disruptions you warn of. I've pasted one example (from Jared) below. I haven't seen any response or acknowledgement from you, to those suggestions, just the repetition of your initial concern. I for one don't understand why you think a scheme such as Jared describes would be either "impossible" or "impractical".

If you are unconvinced about the practicality of such a scheme, it would be great if you could describe why. 

Thanks,

--John

Jared's message:
> On Apr 20, 2017, at 9:40 AM, Jared Mauch <[email-masked]> wrote:
> 
> 
...

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 11:43:17 -0700

Jared,

Also to be clear, I do not question your intentions.  Just the contrary, I understand
the operational issue and the proposal, and I am fine for it to be a BCP.

My question is about the "proposed standard" track and also the mandate for changing
the default behavior in the code.

Regards, -- Enke

---

From: Eric C Rosen <[email-masked]>
Date: Thu, 20 Apr 2017 16:00:39 -0400

On 4/20/2017 2:24 PM, Tony Przygienda wrote:
> Can't miss that food fight ;-)

It did seem to degenerate rapidly into "if you don't agree with my 
proposal, you don't care about security".  ;-(

I agree with Enke that surprising your customers with a change in 
behavior due to altered defaults is generally considered to be a big no-no.

When the customers complain about a change in behavior, it is not 
considered appropriate to respond with "it's not my fault, you should 
have read the release notes", or "it's not my fault that you don't know 
how to troubleshoot BGP", or "it's not my fault that you didn't do your 
due diligence".

Phasing in a change of behavior over several releases is not a practical 
solution, because:

a) Customers will still be surprised when the default behavior finally 
changes, and
b) Many customers won't deploy all the releases anyway.

>
> Having said that, I think this is BCP material at best and if this is 
> a BCP then
>
> i) a "backward compatibility a.k.a which end of the stick is sharp" 
> section is very advisable

I would agree that something more than "figure out how to configure the 
new release to behave like the old release" would be helpful.

> ii) the BCP should describe which customer segment is best served with 

But then operators from different segments would have to get together to 
understand each others' requirements, and they'd have to respect each 
others' opinions as well.   I can't wait to see what happens when the 
"trust nobody" folks get together with the "zeroconfig plug and play" 
folks ;-)

The dilemma is that there is a real security problem in certain 
environments, but the proposed solution seems to have unintended 
side-effects that are problematic.  Then the question becomes whether 
the benefits are worth the cost, and this is not really a question that 
can be resolved by IETF consensus.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 22:19:44 +0200

> I can't wait to see what happens when the "trust nobody" folks get
> together with the "zeroconfig plug and play" folks ;-)

Brilliant analogy !

The crux of the issue is that BGP is used in completely different
scenarios, but completely different users. Some care about Internet some do
not.

For some BGP should be as simple as RIP. For the others it is not even BGP
protocol but BGP policy what is the most critical and important.

It's like cooking big pot of soup .. for some it will always be too spicy
... for others way too mild regardless how much piri-piri you add to it.
That's why restaurants have a concept of menu.

I think it is time to propose proven in other working groups the concept of
profiles for eBGP and perhaps also for iBGP:

*eBGP internet-profile* - will have all the defaults on to make sure
Internet is as secured as possible

*eBGP dc-profile* - will optimize BGP to work in tor-leaf-spine env if
someone needs to use it there

*eBGP ix-profile* - will have set of functionality to work well over NBMA
and auto detect NH liveness across IX

*eBGP home-profile* - will allow to exchange routes and subnets without
much clue about BGP attribiutesm policy and communities

*iBGP maximum-rebustness* - will enable add-paths, best external, enable
PIC etc ...

*iBGP weak-hardware* - will only send and accept one path and slow down
best path runs

etc ..

And all of those profiles will actually differ in the vendor's defaults
optimal to each use case of BGP. And most important it will be backwords
compatible too !

Just a suggestion. If someone is willing to work on draft on it - I can
volunteer to help.

Cheers,
Robert

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 14:03:23 -0700

Hi, John:

It's not like the issues with software update are new :-)

As we all know, there is a complete lack of uniformity (on a global scale) with
the procedures for software update. As a result, a proposal that assumes certain
procedures is bound to be incomplete, and will not work for some.

As Eric summarized,

---
Phasing in a change of behavior over several releases is not a practical solution, because:

a) Customers will still be surprised when the default behavior finally changes, and
b) Many customers won't deploy all the releases anyway.
---

Thanks.  -- Enke

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 23:05:24 +0200

+

c) deployment of new code release in large network takes months if not
years

On Thu, Apr 20, 2017 at 11:03 PM, Enke Chen <[email-masked]> wrote:

> > Various people have provided worked examples of how they think this
> change could be effected without causing the disruptions you warn of. I've
> pasted one example (from Jared) below. I haven't seen any response or
> acknowledgement from you, to those suggestions, just the repetition of your
> initial concern. I for one don't understand why you think a scheme such as
> Jared describes would be either "impossible" or "impractical".
> >
> >>> You can not set "insecure mode" before you reload the OS as current OS
> does not have such knob. Unless you delay the deployment across N releases
> and enforce sequenced upgrade.
> >>
> >> Infact, this is the recommendation that I’ve provided to vendors that
> have expressed concerns.  There are many defaults that have not always been
> displayed, but things like IOS have “show run all” so you can see these.
> >>
> >> Something like the ‘bgp unsafe-ebgp-policy’ could be generated on their
> respective implementations.  I didn’t think that GROW/IDR needed to tell
> implementors this level of how to manage their release, so this does seem
> somewhat out of scope, but a concern I can see needs to be thought about.
> >>
> >>> The only way to prevent massive reachability failure upon reload due
> to complete silent bgp prefix drop is to configure inbound policy for all
> EBGP sessions before the reload and run with new image.
> >>
> >> - Vendors can take the N+1.x and N+2.x release strategy, where in N and
> N+1 they generate their equivalent of IOS-XR and the "bgp
> unsafe-ebgp-policy” policy to prevent their customers from breaking
> >> Most people don’t, and I’ve always suggested to vendors they implement
> some sort of incremental approach to resolving this.
> >>
> >> What worries me is that there is a major incumbent provider who doesn’t
> see this as the serious (and well-documented) security issue that it is for
> those operating large networks.
> >>
> >>> If they do not the troubleshooting of this will be really painful ! CE
> will see EBGP session as UP, will get all the routes and will send his
> routes. CE will have no clue if PE dropped or accepted his routes. Likewise
> on the other end .. Only imagine a network which has 10s of thousands of
> VPN CEs as Bruno mentioned and their provider not following all releases
> CEs are running.
> >>
> >>

---

From: Job Snijders <[email-masked]>
Date: Thu, 20 Apr 2017 21:08:55 +0000

So a change like bgp-reject will take many years to be deployed within a
single network, how can that be reconciled with the perceived "surprise"?

Kind regards,

Job

On Thu, 20 Apr 2017 at 23:05, Robert Raszuk <[email-masked]> wrote:

>> been deployed.
>> > If you are unconvinced about the practicality of such a scheme, it
>> would be great if you could describe why.
>> >
>> > Thanks,
>> >
>> > --John
>> >
>> > Jared's message:
>> >>> You can not set "insecure mode" before you reload the OS as current
>> OS does not have such knob. Unless you delay the deployment across N
>> releases and enforce sequenced upgrade.
>> >>
>> >> Something like the ‘bgp unsafe-ebgp-policy’ could be generated on
>> their respective implementations.  I didn’t think that GROW/IDR needed to
>> tell implementors this level of how to manage their release, so this does
>> seem somewhat out of scope, but a concern I can see needs to be thought
>> about.
>> >>
>> >> - Vendors can take the N+1.x and N+2.x release strategy, where in N
>> and N+1 they generate their equivalent of IOS-XR and the "bgp
>> >> What worries me is that there is a major incumbent provider who
>> doesn’t see this as the serious (and well-documented) security issue that
>> it is for those operating large networks.
>> >>
>> >>> If they do not the troubleshooting of this will be really painful !
>> CE will see EBGP session as UP, will get all the routes and will send his

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 20 Apr 2017 14:08:45 -0700

amen ...

On Thu, Apr 20, 2017 at 1:00 PM, Eric C Rosen <[email-masked]> wrote:

> I agree with Enke that surprising your customers with a change in behavior
> due to altered defaults is generally considered to be a big no-no.
>
> considered appropriate to respond with "it's not my fault, you should have
> read the release notes", or "it's not my fault that you don't know how to
> troubleshoot BGP", or "it's not my fault that you didn't do your due
> diligence".
>
>> Having said that, I think this is BCP material at best and if this is a
>> BCP then
>>
> "trust nobody" folks get together with the "zeroconfig plug and play" folks
> ;-)
>
> side-effects that are problematic.  Then the question becomes whether the
> benefits are worth the cost, and this is not really a question that can be
> resolved by IETF consensus.
>
>
>
>
>
>
>

---

From: John Scudder <[email-masked]>
Date: Thu, 20 Apr 2017 21:12:45 +0000

(as an individual contributor)

> It's not like the issues with software update are new :-)

Yes, exactly. That's why I'm so surprised by this discussion. 

What I still see here is

- on one hand, a worked example showing how an implementor could roll out the functionality without causing heartburn for users. (My paraphrase: expose the default in the configuration. When upgrading old->new, automatically create the corresponding configuration line(s) to configure for legacy behavior.)

- on the other hand, general statements about how this is a hard problem.

I find the specific worked example more convincing. 

--John

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 20 Apr 2017 23:13:55 +0200

Hey Job,

At least I am not seeing that much of a problem for large networks with
bgp-reject. We can collectively go around and educate folks at various NOGs
what is coming well before any vendor publishes code with that change,

The worry is about little guys with one or two routers which never attend
NOGs and which in fact still want to send everything and get default out
from upstream. I am feeling a bit sorry to expose them to such pain
regardless if they use cisco, quagga, ubiquiti, goBGP, snaproute ... you
name it.

//R

On Thu, Apr 20, 2017 at 11:08 PM, Job Snijders <[email-masked]> wrote:

>>> >> Most people don’t, and I’ve always suggested to vendors they
>>> implement some sort of incremental approach to resolving this.
>>> >>

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 14:25:29 -0700

John,

On 4/20/17 2:12 PM, John Scudder wrote:
> - on one hand, a worked example showing how an implementor could roll out the
>   functionality without causing heartburn for users. (My paraphrase: expose the
>   default in the configuration. When upgrading old->new, automatically create
>   the corresponding configuration line(s) to configure for legacy behavior.)

Change of software may involve both upgrade and downgrade and combination (e.g.,
when a serious issue is seen).

When the software is downgraded, the config may not be recognized and may be
lost.

-- Enke

---

From: Eric C Rosen <[email-masked]>
Date: Thu, 20 Apr 2017 17:49:34 -0400

On 4/20/2017 5:08 PM, Job Snijders wrote:
> So a change like bgp-reject will take many years to be deployed within 
> a single network, how can that be reconciled with the perceived 
> "surprise"?
>

The surprise comes when the first router with the new defaults is deployed.

The fact that one or two intermediate releases had been deployed during 
the previous several years doesn't lessen the surprise.

This isn't a problem for the folks who want the defaults changed and who 
understand all the issues, it's only a problem for everyone else.

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 20 Apr 2017 20:08:09 -0400

(still as an individual contributor)

> On Apr 20, 2017, at 5:25 PM, Enke Chen <[email-masked]> wrote:
> 

Sure. What of it? Presumably the downgraded software has the legacy behavior. When and if you re-upgrade it, the logic quoted above to emulate the legacy behavior is re-applied just like with the first upgrade. The net result is legacy behavior continues through upgrade and downgrade. (The next step in this argument is "but if you always maintain legacy behavior, what's the point?" but that's already been answered -- by Jared, IIRC -- much earlier in the thread.)

--John

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 20:40:28 -0400

On Thu, Apr 20, 2017 at 04:00:39PM -0400, Eric C Rosen wrote:
> It did seem to degenerate rapidly into "if you don't agree with my proposal,
> you don't care about security".  ;-(
> 

	I'm sure you are surprised that I might disagree here.

	I've experienced many software defects with disastrous side-effects,
this has not stopped the vendors from releasing junk code and people
running said-junk :-)

	What I see here is a complete capitulation of "this is hard,
lets go shopping".

> When the customers complain about a change in behavior, it is not considered
> appropriate to respond with "it's not my fault, you should have read the
> release notes", or "it's not my fault that you don't know how to

	Customers who do extensive testing still find issues that
are not found in labs, their own testing and limited deployments, does this
mean all software developers are out of a job as it's not worthwhile
to invest in development?

> Phasing in a change of behavior over several releases is not a practical

I must disagre.

> a) Customers will still be surprised when the default behavior finally

	Customers who do no testing will always be surprised.  Customers who
do hopefully have a process for things, and while annoying may be less surprised.

	I recall when a vendor made /31's illegal on ethernet interfaces,
testing was done in my lab, but not with /31s, there was a gap.  This caused an
outage as it wiped the infrastructure links.  This wasn't release noted,
the vendor was briefly flogged and we moved on to more pressing defects.  This
is very similar to what John  and others suggested, you downgrade and try again
later.

> b) Many customers won't deploy all the releases anyway.

	I'm not sure the relevance here?  Some people upgrade often, some people
upgrade once a year, some never.  aka, Sky still blue.

> I would agree that something more than "figure out how to configure the new
> release to behave like the old release" would be helpful.

	Text welcome.

> others' opinions as well.   I can't wait to see what happens when the "trust
> nobody" folks get together with the "zeroconfig plug and play" folks ;-)
> 

	I disagree here, we can say "routing leaks are a problem, we should
minimize them".  This has been documented by IETF consensus in RFC7908.

	It may well be the various camps can't agree, because there are:

a) internet-at-large
   (who cares about the global reachability?  i do!)

b) datacenter
   (what i do on my own turf is my own problem! ztp ftw!)

c) enterprise
   (i'm paid to not rock the boat)

	What I'm hoping we can find consensus on is this is a problem
worthwhile fixing and IDR/GROW/IETF care enough to fix it vs the taylor swift
response of "we are never ever getting back together", "this is unsolvable",
"my default behavior is right", "i am a granite rock".

	To those who say BCP-38 is a failure because it's not 100%, you are
right and wrong.  Significant parts of networks are BCP-38 enabled, some parts
are not.  Losing sight of the goal is easy when you're throwing stones.

	Text mitigating what your concerns are is welcome to the authors alias
or the IDR/GROW lists.

	(note to self, unsubscribe from the other ietf lists you are not reading).

	- Jared

---

From: Jared Mauch <[email-masked]>
Date: Thu, 20 Apr 2017 20:46:07 -0400

On Thu, Apr 20, 2017 at 05:49:34PM -0400, Eric C Rosen wrote:
> The fact that one or two intermediate releases had been deployed during the
> previous several years doesn't lessen the surprise.
> 

	Thankfully vendors are perfect :-)

	I understand your problem, but then what is the acceptable time period,
hardware life cycle of 7 years sound good enough?  Could you say by 2024 all new
releases could be compliant with a new operational practice?

	I've been actively watching the leaks for nearly a decade now, another 7
years would be quite sad, but at least better than never.  I'd rather try something
because repeating our mistakes for the next decade is unpalatable to me.

	- Jared

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 17:53:30 -0700

As I understand, the scheme requires the config to be generated in the intermediate
releases.  Once you lose the config and then try to upgrade (and skipping the
intermediate releases), we are back to the original state and problem.

So it is the same issue, that is, upgrading to a new release without going
through the intermediate releases that generate the configs.  The routes
accepted by the old release would be rejected by the new release.

-- Enke

On 4/20/17 5:08 PM, John G. Scudder wrote:
> Sure. What of it? Presumably the downgraded software has the legacy behavior.
> When and if you re-upgrade it, the logic quoted above to emulate the legacy
> behavior is re-applied just like with the first upgrade. The net result is
> legacy behavior continues through upgrade and downgrade. (The next step in
> this argument is "but if you always maintain legacy behavior, what's the point?"
> but that's already been answered -- by Jared, IIRC -- much earlier in the thread.)
> 
> --John
>

---

From: John Scudder <[email-masked]>
Date: Fri, 21 Apr 2017 01:02:17 +0000

(Do I need to keep saying I am writing as an individual contributor? Better safe than sorry I suppose.)

Nah. Generate it in the new releases if you see a config that came from an older version. (You can tell this, among other ways, by noticing that the required new line of configuration is not present).

There is no intermediate version.

Of course, endless variant schemes are possible. This is merely a thought experiment to prove the problem is neither intractable, nor even particularly difficult.

--John

> On Apr 20, 2017, at 8:53 PM, Enke Chen <[email-masked]> wrote:
>

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 18:08:51 -0700

So *all* the new releases would generate "permit all" for any neighbor that does
not have an inbound policy?  I am confused - how is this "permit all" different
from today?

-- Enke

On 4/20/17 6:02 PM, John Scudder wrote:
> Nah. Generate it in the new releases if you see a config that came from an older version.
> (You can tell this, among other ways, by noticing that the required new line of configuration
> is not present).
>

---

From: Enke Chen <[email-masked]>
Date: Thu, 20 Apr 2017 18:13:36 -0700

John,

You have many hats :-) and most of us have only one :-(

-- Enke

---

From: John Scudder <[email-masked]>
Date: Fri, 21 Apr 2017 01:29:09 +0000

#include individual contributor disclaimer

> On Apr 20, 2017, at 9:08 PM, Enke Chen <[email-masked]> wrote:
> 
> not have an inbound policy?  

Something like that. We are into minutiae here but in my imagined case it's not a per-peer option, it's a global "emulate legacy defaults" option. (I think Jared called it "set bgp insecure-mode on" or something similarly provocative.)

The global is only generated if (a) there is an existing config being upgraded and (b) it doesn't already have the command present (so, the way you turn it off is... configure it off). 

> I am confused - how is this "permit all" different
> from today?

It's not. That's the point. I had hoped to avoid rehashing this since it was already covered in the thread but as I recall the claimed advantages vs. "do nothing" included

- there are no surprises for legacy deployments as you've been warning of, since as you say the black-box behavior is unchanged,
- at least the default is explicit in the config so if you call it "insecure-mode" or something, maybe people will have incentive to flip it from "on" to "off",
- new deployments (where a legacy config is not being upgraded) would start with the default "off" -- this is probably the most important point. 

--John

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 09:25:17 +0200

All,

The entire new proposal is to help operate eBGP sessions. And this is
perhaps even cool for Internet sessions.

But I think silent route drops or making them invalid for best path ..
those options are not cool.

I would very much prefer to keep all eBGP sessions down ... not even send
out OPEN where such policy is not in place in new release or when someone
removes auto configured "bgp insecure mode". This is easy to troubleshoot,
to notice that something is not right and fix by anyone immediately or roll
back to previous release and come back to vendor with complain ;).

Having sessions to be up normally and getting silent drops on the peers
side where you can not login and check I find hard to accept as reasons for
it can be counted in 10s if not 100s.

//R.

On Apr 21, 2017 03:29, "John Scudder" <[email-masked]> wrote:

> Something like that. We are into minutiae here but in my imagined case
> it's not a per-peer option, it's a global "emulate legacy defaults" option.
> (I think Jared called it "set bgp insecure-mode on" or something similarly
> provocative.)
>
> The global is only generated if (a) there is an existing config being
> upgraded and (b) it doesn't already have the command present (so, the way
> you turn it off is... configure it off).
>
> It's not. That's the point. I had hoped to avoid rehashing this since it
> was already covered in the thread but as I recall the claimed advantages
> vs. "do nothing" included
>
> - there are no surprises for legacy deployments as you've been warning of,
> since as you say the black-box behavior is unchanged,
> - at least the default is explicit in the config so if you call it
> "insecure-mode" or something, maybe people will have incentive to flip it
> from "on" to "off",
> - new deployments (where a legacy config is not being upgraded) would
> start with the default "off" -- this is probably the most important point.
>
> --John
>

---

From: <[email-masked]>
Date: Fri, 21 Apr 2017 07:32:29 +0000

> From: John Scudder  Sent: Thursday, April 20, 2017 11:13 PM
> 
 > functionality without causing heartburn for users. (My paraphrase: expose the default in
 > the configuration. When upgrading old->new, automatically create the corresponding
 > configuration line(s) to configure for legacy behavior.)
 > 

Good point. Thanks for the useful contribution to the debate.

a) Regarding this specific point, could the draft refer to this point. e.g. add this requirement (if this is a requirement draft, which it looks like it when reading the first sentence of section 2) or add the solution (space) (if the draft define a solution).
e.g. "When the BGP speaker is upgraded to those new default behavior, existing BGP sessions, configured using the old default, must be unaffected".
Or whatever text.

That is indeed the most significant deployment impact, so it's good that it can be addressed/included, rather than silently ignored.

b) Still, this draft introduce new deployment impact to all network operators using EBGP, including the ones not concerned with this problem space, as provisioning tools need to be updated prior to this deployment, otherwise provisioning would fails. Plus silently fails as the session is kept up, silently ignoring the route with no feedback to the EBGP peer). Plus for each plateform, they need to selectively handle both configurations depending on the software version of this plateform.

Regards,
--Bruno

---

From: <[email-masked]>
Date: Fri, 21 Apr 2017 08:20:47 +0000

> From: Eric C Rosen  > Sent: Thursday, April 20, 2017 10:01 PM
> 

Good summary, thanks.

1) As already asked, could the draft states what the problem it aims at solving?
In particular,
- it does not solve the general route leak problem, nor the 3 problems indicated in the first section of the introduction.
- on the list, one author expressed a different, more focused/pragmatic, goal
" The problem is that there are some platforms which _immediately_
activate a line of configuration after you press enter, and a BGP
session may already come up before you get to the configuration line
which restricts what should be announced or accepted."

2) Can the draft document the side-effects, e.g. in a manageability section.

Without those 2 points, IMHO the draft is misleading for the reader/reviewer/voter/user. This trade-off analysis is currently absent in the document up to the point that those drawbacks have not been raised during RTG-DIR and RTG-OPS review. So it's probably fair to assume that this may also have missed by some other IETF contributors.

3) Once the problem that we want to solve has been clarified (cf 1) , then we can start reviewing the solution.
In particular, if the problem is indeed:
" The problem is that there are some platforms which _immediately_
activate a line of configuration after you press enter, and a BGP
session may already come up before you get to the configuration line
which restricts what should be announced or accepted."

Then, there might be other solutions. e.g.
- fixing this CLI issue on those plateforms. In particular, no need to also impact the netconf provisioning which has not this issue, or the implementations which do not have such CLI issue.
- use a 2 step provisioning on both EBGP peers:
   - on the sending side, use a trick for avoid the session to come up until the BGP configuration is complete enough . e.g. interface shut, use the wrong authentication key/TTL hack
   - on the receiving side, use a trick to avoid the session to successfully come up or to accept the routes. Then after allowing for a reasonable time to let the sender finish its config, restore the correct configuration. Obviously this can be automated by a "controller", a local feature, a local event script.
This 2 step provisioning has the benefit of limiting the cost of the solution to the space where the problem is. (vs asking everyone to bear the cost, for a benefit of a few, whatever important is this use case)
- I sure others may find more creative solutions.

4) Regarding the specific solution proposed in the draft,
-  why has it been prefer to silently drop the route, with will surprise people and definitely the EBGP peer, rather than bringing done the BGP session with a reasonable error (sub)code / text to help diagnose the problem.

- the 3rd bullet is debatable to me
"   o  A BGP speaker SHOULD fall back to an "import nothing" and "export nothing" mode following failure of internal components, such as a policy engine."
This is an error handling situation. IMHO https://tools.ietf.org/html/draft-ietf-grow-ops-reqs-for-bgp-error-handling-07 had a better analysis on this point. e.g. at the minimum, rather than calling for silently dropping the route, I would call for closing the BGP session, possibly trying to restart it. Plus if this BGP internal failure also impact IBGP session, we have a bigger problem.

- it's not clear whether this draft is a requirement draft or a solution draft. As per the first sentence of section 2 and its further text, it looks like a requirement draft. But in this case, the requirements should be solution agnostic. Which is definitely not the case. (e.g. above comments are solutions specific)

Thanks,
Regards,
-- Bruno

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 10:46:38 +0200

On Fri, Apr 21, 2017 at 07:32:29AM +0000, [email-masked] wrote:
>  > - on one hand, a worked example showing how an implementor could
>  > roll out the functionality without causing heartburn for users. (My
>  > paraphrase: expose the default in the configuration. When upgrading
>  > old->new, automatically create the corresponding configuration
>  > line(s) to configure for legacy behavior.)
>  > 
> a) Regarding this specific point, could the draft refer to this point.
> e.g. add this requirement (if this is a requirement draft, which it
> looks like it when reading the first sentence of section 2) or add the
> solution (space) (if the draft define a solution).
> e.g. "When the BGP speaker is upgraded to those new default behavior,
> existing BGP sessions, configured using the old default, must be
> unaffected". Or whatever text.

You are adding complexity and increasing inconsistency rather then
reducing it.

> That is indeed the most significant deployment impact, so it's good
> that it can be addressed/included, rather than silently ignored.
> 
> b) Still, this draft introduce new deployment impact to all network
> operators using EBGP, including the ones not concerned with this
> problem space, as provisioning tools need to be updated prior to this
> deployment, otherwise provisioning would fails. Plus silently fails as
> the session is kept up, silently ignoring the route with no feedback
> to the EBGP peer). Plus for each plateform, they need to selectively
> handle both configurations depending on the software version of this
> plateform.

So, going forward, any IDR proposal that requires a change to any
provision system is a no-go?

For the record: NTT has many conditional clauses in their configuration
generation system to deal with configuration differences between
different versions of the software on the same platforms. This is
common. I do not believe you or anyone else operate your networks
without the accepting that different versions may have for example
different syntax.

I've already stated how the failure scenario can only occur after a
missteps in the process, and that outages are likely to be remedied
swiftly and permanently. I've also contributed a case where bgp-reject
actually reduces the duration of outages that would otherwise be
caused and prolonged if an insecure mode of operation is used.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 11:01:45 +0200

Hi Bruno,

You create a false dilemma stating that it is not clear to you whether
this is a "requirement draft or a solution draft", and that those need
to be separate. 

Did you review Alvaro's suggested changes which prompted the third IDR
consultation on this topic? They are viewable here:

    http://htmlpreview.github.io/?https://github.com/jaredmauch/draft-mauch-bgp-reject/blob/alvaro/draft-ietf-grow-bgp-reject-06-from-5.diff.html

I'm disappointed that you've taken a CLI example I've provided as the
sole driver behind this effort. I did not mean to offer the CLI example
as an exhaustive list of reasons.

Kind regards,

Job

ps. For additional advise on how to derail this effort, let us please
take the ideas offered in
http://www.businessinsider.com/oss-manual-sabotage-productivity-2015-11?international=true&r=US&IR=T
also in consideration.

---

From: <[email-masked]>
Date: Fri, 21 Apr 2017 09:18:24 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Friday, April 21, 2017 10:47 AM
> 

:s/adding/dealing with the

I guess that this means a "No" to my request.

 > provision system is a no-go?
 
It's not about a no-go. It's about understanding and considering the tradeoffs.

 > without the accepting that different versions may have for example

I'm not saying that this is not possible. I'm saying that there is an impact, with associated cost and delay.
If possible, I'd prefer a solution which preserve the benefits while reducing the impacts. This requires that we all agree on the problems that we want to solve and the consequences of a proposed solution.

At the very minimum, 
"  o  A BGP speaker MAY provide a configuration option to disable the preceding behaviors, but it MUST implement them by default."
:s/MAY/MUST

But IMHO, we could discuss the "MUST implement them by default". As basically, the document is saying that the people facing the issue with their CLI, cannot even be asked to enable a one line knob in their configuration, while the document cannot even acknowledge and take into account the cost added on all other EBGP users.

Kind regards,
--Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 11:35:06 +0200

​Hi Job,​

>     http://htmlpreview.github.io/?https://github.com/jaredmauch/
> draft-mauch-bgp-reject/blob/alvaro/draft-ietf-grow-bgp-
> reject-06-from-5.diff.html
>
>
Do we really have consensus among all authors of this document (leave alone
rest of IETF) that it should apply to all AFI/SAFIs ?

I personnally would recommend to apply it to only 1/1 and 2/1.

And actually even take those SAFIs your claim that it helps case of
sessions being dropped is very implementation specific ... some
implementation when receiving routes to VRF from CE may first check the
maximum-prefix number as expressed in "neighbor 10.1.1.1 maximum-prefix 30"
then check per VRF then apply say BGP prefix-list.

Does this RFC also enforces that BGP route-map or policy should be the
first consideration in BGP inbound processing ?

explicit configuration of a BGP import and export policy for any
External BGP (EBGP) session such as customers, peers, or
confederation boundaries
*​​for all enabled address families*. When this

- - -

Last ... what's your take on multiple comments regarding build in silency
of this ?

See session will be established just fine ... even incoming prefix count
will be incremented just fine.

Last time I checked until you display bgp table there is no separate
counter in any implementation to indicate in BGP summary how many out of
received paths were best path eligible.

Thx,
r.

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 11:58:39 +0200

> It's not about a no-go. It's about understanding and considering the

Bruno, through this thread I have learned:

    - Release notes are not read, and it is unreasonable to expect
      people to read them.
    - Software will not be tested prior to deployment, not even to see
      whether it boots.
    - We cannot expected staggered software deployments, people will
      upgrade all their BGP speakers at the same time (again, without
      testing the software).
    - Any change to a provisioning system is an insurmountable
      imposition.

I'm sure you appreciate how these new insights will affect all future
IDR work. I assure you that if such weak rethoric continues to be
admitted as valid justifications for lethargy, this will affect IDR's
productivty the coming years.

If you want to play the game of 'tradeoffs have to be made', I have not
seen any appreciation in this thread for the cost on the Internet as a
whole resulting from insecure defaults. Robert Raszuk even went as far
to argue that we'd be doing the "little guys" (surely that was not meant
in a belligerent way) a favor by allowing insecure defaults to persist. 

I theorize that for instance this outage was the result of a 'fail open'
rather then 'fail closed' (as proposed in bgp-reject) implmentation
choice. See https://www.theregister.co.uk/2015/06/12/level_3_down_after_routing_through_malaysia_like_idiots/
or https://bgpmon.net/massive-route-leak-cause-internet-slowdown/

Outages like these affect everyone, whether they were a directly
involved party or indirectly involved. Did you notice this event in your
own network? Or perhaps you did notice it because payment terminals in
some countries stopped working?

The BGP Default-Free Zone is composed of roughly 55,000 autonomous
systems operated by as many organisations, who are densily
interconnected with each other through milions of EBGP sessions. When DC
equipment is connected to Internet, or when a CLI-style makes accidents
easy, or when a lack of education results in a common misconfiguration,
there should be checks and balances in place to dampen the negative
effects on the Internet as a whole. The Internet Engineering Task Force
(notice the 'Internet' in IETF) has a responsiblity to promote and
define safe and secure default behaviours.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 12:11:16 +0200

On Fri, Apr 21, 2017 at 11:35:06AM +0200, Robert Raszuk wrote:
> Do we really have consensus among all authors of this document (leave
> alone rest of IETF) that it should apply to all AFI/SAFIs ?

Feel free to double check with the GROW chairs or the assigned OPS Area
Director.

> Last ... what's your take on multiple comments regarding build in
> silency of this ?
> 
> See session will be established just fine ... even incoming prefix
> count will be incremented just fine.

The perceived 'silency' is a misnomer. Vendors are free implement
feedback mechanisms (such as a syslog or terminal warning: "you did not
configure a policy"), and some already provide feedback like this.
Nothing in this draft precludes vendors from ensuring the operation of
their BGP implementations a pleasant experience.

No feedback is required for the other side of the EBGP session, after
all the ownership of the change event lays with the side that upgrades
their software (same applies when the 'changing side' removes a policy,
or applies a deny-all policy, or any other standard operating
procedure).

You also fail to acknowledge that today's deployed BGP speaking software
presents a myriad of behaviours: some already follow bgp-reject, some
apply the concept in one direction and some operate in an promiscuous
insecure mode. This draft provides guidance to unify those behaviours
and provide a safe, consistent experience across multiple
implementations. 

> counter in any implementation to indicate in BGP summary how many out
> of received paths were best path eligible.

I encourage you to continue your studies on how today's routers work and
what counters are available.

Kind regards,

Job

---

From: <[email-masked]>
Date: Fri, 21 Apr 2017 11:58:18 +0000

Hi Job,

> From: Job Snijders [mailto:[email-masked]]  > Sent: Friday, April 21, 2017 11:02 AM
> 
 > Hi Bruno,
 > 
 > this is a "requirement draft or a solution draft", 

That editorial point should be easy to address.

> and that those need to be separate.

I've not said that.

 >     http://htmlpreview.github.io/?https://github.com/jaredmauch/draft-mauch-bgp-
 > reject/blob/alvaro/draft-ietf-grow-bgp-reject-06-from-5.diff.html

I was discussing the current version of the draft and comments sent on the mailing list (e.g less than 24 hours ago, one author commented that it does not believe that this document should be changed to update RFC 4271: " I believe that Alvaro is wrong in the presumption
this document updates 4271. ".
I was not aware of this private repository. Indeed, this candidate version is significantly different than the public one.
 
 > as an exhaustive list of reasons.

That this incorrect: I've commented on this driver, plus the ones indicated in the draft. What's missing has been authors' answer for those requests for clarification.
For the third time, could authors please document in the draft the drivers and the goal(s) that this document aims at achieving. Then we'll be able to take into account all drivers, plus authors will get IDR attention to their operational feedback. Should be win-win.

-- Bruno

 > > Without those 2 points, IMHO the draft is misleading for the reader/reviewer/voter/user.
 > This trade-off analysis is currently absent in the document up to the point that those
 > drawbacks have not been raised during RTG-DIR and RTG-OPS review. So it's probably fair to
 > assume that this may also have missed by some other IETF contributors.
 > >
 > > - fixing this CLI issue on those plateforms. In particular, no need to also impact the netconf
 > provisioning which has not this issue, or the implementations which do not have such CLI
 > issue.
 > >    - on the sending side, use a trick for avoid the session to come up until the BGP
 > configuration is complete enough . e.g. interface shut, use the wrong authentication
 > key/TTL hack
 > >    - on the receiving side, use a trick to avoid the session to successfully come up or to
 > accept the routes. Then after allowing for a reasonable time to let the sender finish its
 > config, restore the correct configuration. Obviously this can be automated by a "controller",
 > a local feature, a local event script.
 > > This 2 step provisioning has the benefit of limiting the cost of the solution to the space
 > where the problem is. (vs asking everyone to bear the cost, for a benefit of a few, whatever
 > important is this use case)
 > > -  why has it been prefer to silently drop the route, with will surprise people and definitely
 > the EBGP peer, rather than bringing done the BGP session with a reasonable error
 > (sub)code / text to help diagnose the problem.
 > >
 > > "   o  A BGP speaker SHOULD fall back to an "import nothing" and "export nothing" mode
 > following failure of internal components, such as a policy engine."
 > > This is an error handling situation. IMHO https://tools.ietf.org/html/draft-ietf-grow-ops-
 > reqs-for-bgp-error-handling-07 had a better analysis on this point. e.g. at the minimum,
 > rather than calling for silently dropping the route, I would call for closing the BGP session,
 > possibly trying to restart it. Plus if this BGP internal failure also impact IBGP session, we have
 > a bigger problem.
 > >
 > > - it's not clear whether this draft is a requirement draft or a solution draft. As per the first
 > sentence of section 2 and its further text, it looks like a requirement draft. But in this case,
 > the requirements should be solution agnostic. Which is definitely not the case. (e.g. above
 > comments are solutions specific)
 > >
 > > Thanks,
 > > Regards,
 > > -- Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 14:33:22 +0200

Job,

Let's face it ... your proposal in spite of all claims you are making does
nothing to prevent outages or route leaks.

Your examples of "accept all" or "send all" are clear prove of it.

Forcing any policy != forcing correct policy.

If someone cares about Internet stability he will implement correct policy
today without being forced by a vendor. If someone care less he will send
all and accept all just line it is today.

So let's not try to sell/sneak someting here under completely wrong label.
All this draft is about it is forcefull education for those customers who
do not know that bgp policy exist.

Kind regards,
R.

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 14:40:11 +0200

On Fri, Apr 21, 2017 at 11:58:18AM +0000, [email-masked] wrote:
>  > You create a false dilemma stating that it is not clear to you
>  > whether this is a "requirement draft or a solution draft", 
> 

The use of the word 'or' in your sentence """it's not clear whether this
draft is a requirement draft or a solution draft.""" led me to believe
that you assert the two are mutually exclusive. Furthermore, from the
context it appears that you recommend to abandon 'bgp-reject', start
over with a requirements document, and then see what happens. If this is
not the case please elaborate, and accept my apologies for interpreting
it as I did.

> I was discussing the current version of the draft and comments sent on
> the mailing list (e.g less than 24 hours ago, one author commented
> that it does not believe that this document should be changed to
> update RFC 4271: " I believe that Alvaro is wrong in the presumption
> I was not aware of this private repository. Indeed, this candidate
> version is significantly different than the public one.

The link to the publicly available private repository was meant as a
convenience to you and the WG. It is a verbatim repetition of Alvaro's
suggestions which were emailed to idr@ on April 18th. I hoped to make
Alvaro's comments easier to read by creating the HTML rfcdiff. Did you
perhaps miss that message?

    https://mailarchive.ietf.org/arch/msg/idr/gU6sk8yrC3uF7aLERwnDh2cuFP4

>  > I'm disappointed that you've taken a CLI example I've provided as
>  > the sole driver behind this effort. I did not mean to offer the CLI
>  > example as an exhaustive list of reasons.
> 
> That this incorrect: I've commented on this driver, plus the ones
> indicated in the draft. What's missing has been authors' answer for
> those requests for clarification.  For the third time, could authors
> please document in the draft the drivers and the goal(s) that this
> document aims at achieving. Then we'll be able to take into account
> all drivers, plus authors will get IDR attention to their operational
> feedback. Should be win-win.

Please review the Introduction section of draft-ietf-grow-bgp-reject-05.

Should you fail to identify drivers and goals in that section, can you
elaborate what it is you do read in that section? 

I find it hard to believe that the Introduction passed through SECDIR,
RTGDIR, GENART, OPSDIR, and GROW Last Call, and was clear to those
involved, yet you indicate that drivers & goals are not clear, I am not
sure what I can do to further your understanding in that regard.

Kind regards,

Job

---

From: <[email-masked]>
Date: Fri, 21 Apr 2017 14:32:01 +0000

Job,

 > From: Job Snijders [mailto:[email-masked]]  > Sent: Friday, April 21, 2017 2:40 PM

Thanks for the clarification. 
On my side, I don't think that there is a need for a dedicated problem statement document. IMHO the place for this problem statement is in the Introduction section of you document.
 
 >     https://mailarchive.ietf.org/arch/msg/idr/gU6sk8yrC3uF7aLERwnDh2cuFP4
 
I had read Alvaro's email.
What I had missed is the link to the private repository, and then its meaning. I had though (apparently incorrectly, sorry for this) that this were the candidate -06 that authors were working on. i.e. the latest "current" status of the draft. From your above clarification, this is not, and -05 is the latest version.
 
 > elaborate what it is you do read in that section?

I had tried in https://mailarchive.ietf.org/arch/msg/idr/9Qn2bHQw1QIruWMCJIoEN0Z-qC4 but can try again below

"   There are BGP routing security issues that need to be addressed to make the Internet more stable."
Agreed, but not very specific. At least this does not present the specific points that this document wants to address/solve. (believe me, I wish you would address all BGP routing security issues but we probably do not want to put the bar too high, as it would be too hard to succeed)

"  Route leaks [RFC7908] are part of the  problem, but software defects or operator misconfigurations can  contribute too."
- I agree that route leak is an important issue. But they seem to be already worked on by draft-ymbk-idr-bgp-open-policy or draft-ietf-idr-route-leak-detection-mitigation. Plus this document is not a general solution to route leak, so this is also not the point that this document wants to address (because otherwise, it failed)
- Software defect won't probably be addressed by this proposal.
- Misconfiguration won't probably be addressed by this proposal. In particular no effort is made to ensure that the export/import policy is good/correct.

"   Many deployed BGP speakers send and accept any and all route
   announcements between their BGP neighbors by default.  This practice
   dates back to the early days of the Internet, where operators were
   permissive in sending routing information to allow all networks to
   reach each other.  As the Internet has become more densely
   interconnected, the risk of a misbehaving BGP speaker poses
   significant risks to Internet routing."

Ok, this is a statement of facts. But:
All BGP speakers requires configuration to establish an EBGP session. When doing this configuration, the operator needs to apply the right policy.
But this draft is not helping to solve "the right" policy issue. All it requires is an additional configuration key word. This is an improvement but:
- This does not cover configuration mistakes.
- An operator is likely to blindly copy paste the new key word (enable policy "no filtering")  as part of configuring the session. In which case the gain would be minimal
 
It's also not clear to me if we can assume that the operator is a reasonable guy which wants to do the good thing, but failed because of a mistake or an improper configuration tool; or if this operator just does not give a damn.
IMHO, given that the guy/company is an operator doing this to get money, I would personally opt for the former. In which case, he could be asked and should be happy to add one single configuration line to enable those new default behaviors. This alone would address my concern. In addition, as this document requires a software upgrade, one could signal this new behavior in the BGP Open, such that you could check by yourself.

"   This specification intends to improve this situation by requiring the
   explicit configuration of a BGP import and export policy for any
   External BGP (EBGP) session such as customers, peers, or
   confederation boundaries for all enabled address families.  When this
   solution is implemented, BGP speakers do not accept or send routes
   without policies configured on EBGP sessions."

That is not part of the problem statement but a summary of the solution. (yet, a useful summary as part of the Introduction)

 > sure what I can do to further your understanding in that regard.

I find it equally hard that nobody complains on the impact/cost in term of existing EBGP configuration missing the new required policy, and impact on service provisioning tools (while they are notoriously heavy, expensive and not very flexible).
Plus I'm not saying that this document has no benefit. I'm saying that it also has cost. And by carefully considering both, we could probably find a tradeoff satisfying all parties (that's my optimistic side, but just like anybody I can be wrong)

Given my vision of the cost/benefit, in term of solution I think that the below change would be reasonable tradeoff. At least it would address my main concern:

OLD:  A BGP speaker MAY provide a configuration option to disable the preceding behaviors, but it MUST implement them by default.
NEW: A BGP speaker MUST provide a configuration option to enable the preceding behaviors. Until network/service provisioning are upgraded, it SHOULD be disabled by default. Such provisioning tools should be upgraded soon is order to prepare for the future new default behavior.

Optionally, on the solution side:
- I would also support the addition of a BGP capability to signal whether this behavior is enabled or not. This would allow an EBGP peer to reject the EBGP session if the configuration is not acceptable to them. But I can live without.
- I would prefer that the BGP session be closed, rather than keeping it up and not considering the received routes/not sending routes.

Thanks,
Kind regards,
--Bruno

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 17:19:28 +0200

One final point ...

> - An operator is likely to blindly copy paste the new key word (enable
policy
> "no filtering")  as part of configuring the session. In which case the
gain would
> be minimal

Exactly !

The draft assumes that there are operators managing those 55000 ASes who
already do know today what to put in their BGP policy and they are not
doing
it only because they are lazy and their vendor does not enforce them to do
it.

Are we really that bad in Internet NOCs ? Do we need configuration
enforcement
and RFCs like this ?

Wouldn't it be a much better to instead share a pointer to Best Practices
for EBGP
Policy Configuration wiki and even include such URL in deployment section
of the draft
under discussion ?

Thx,
r.

On Fri, Apr 21, 2017 at 4:32 PM, <[email-masked]> wrote:

> On my side, I don't think that there is a need for a dedicated problem
> statement document. IMHO the place for this problem statement is in the
> Introduction section of you document.
>
> draft-mauch-bgp-
> What I had missed is the link to the private repository, and then its
> meaning. I had though (apparently incorrectly, sorry for this) that this
> were the candidate -06 that authors were working on. i.e. the latest
> "current" status of the draft. From your above clarification, this is not,
> and -05 is the latest version.
>
> I had tried in https://mailarchive.ietf.org/arch/msg/idr/
> 9Qn2bHQw1QIruWMCJIoEN0Z-qC4 but can try again below
>
>
> make the Internet more stable."
> Agreed, but not very specific. At least this does not present the specific
> points that this document wants to address/solve. (believe me, I wish you
> would address all BGP routing security issues but we probably do not want
> to put the bar too high, as it would be too hard to succeed)
>
> "  Route leaks [RFC7908] are part of the  problem, but software defects or
> operator misconfigurations can  contribute too."
> - I agree that route leak is an important issue. But they seem to be
> already worked on by draft-ymbk-idr-bgp-open-policy or
> draft-ietf-idr-route-leak-detection-mitigation. Plus this document is not
> a general solution to route leak, so this is also not the point that this
> document wants to address (because otherwise, it failed)
> - Misconfiguration won't probably be addressed by this proposal. In
> particular no effort is made to ensure that the export/import policy is
> good/correct.
>
> All BGP speakers requires configuration to establish an EBGP session. When
> doing this configuration, the operator needs to apply the right policy.
> But this draft is not helping to solve "the right" policy issue. All it
> requires is an additional configuration key word. This is an improvement
> but:
> - An operator is likely to blindly copy paste the new key word (enable
> policy "no filtering")  as part of configuring the session. In which case
> the gain would be minimal
>
> It's also not clear to me if we can assume that the operator is a
> reasonable guy which wants to do the good thing, but failed because of a
> mistake or an improper configuration tool; or if this operator just does
> not give a damn.
> IMHO, given that the guy/company is an operator doing this to get money, I
> would personally opt for the former. In which case, he could be asked and
> should be happy to add one single configuration line to enable those new
> default behaviors. This alone would address my concern. In addition, as
> this document requires a software upgrade, one could signal this new
> behavior in the BGP Open, such that you could check by yourself.
>
>
> That is not part of the problem statement but a summary of the solution.
> (yet, a useful summary as part of the Introduction)
>
>
> I find it equally hard that nobody complains on the impact/cost in term of
> existing EBGP configuration missing the new required policy, and impact on
> service provisioning tools (while they are notoriously heavy, expensive and
> not very flexible).
> Plus I'm not saying that this document has no benefit. I'm saying that it
> also has cost. And by carefully considering both, we could probably find a
> tradeoff satisfying all parties (that's my optimistic side, but just like
> anybody I can be wrong)
>
>
> Given my vision of the cost/benefit, in term of solution I think that the
> below change would be reasonable tradeoff. At least it would address my
> main concern:
>
> OLD:  A BGP speaker MAY provide a configuration option to disable the
> preceding behaviors, but it MUST implement them by default.
> NEW: A BGP speaker MUST provide a configuration option to enable the
> preceding behaviors. Until network/service provisioning are upgraded, it
> SHOULD be disabled by default. Such provisioning tools should be upgraded
> soon is order to prepare for the future new default behavior.
>
> - I would also support the addition of a BGP capability to signal whether
> this behavior is enabled or not. This would allow an EBGP peer to reject
> the EBGP session if the configuration is not acceptable to them. But I can
> live without.
> - I would prefer that the BGP session be closed, rather than keeping it up
> and not considering the received routes/not sending routes.
>
> Thanks,
> Kind regards,
> --Bruno
>
>  > Kind regards,
>  >
>  > Job
>
> ____________________________________________________________
> _____________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;

---

From: Gert Doering <[email-masked]>
Date: Fri, 21 Apr 2017 17:37:41 +0200

Hi,

On Fri, Apr 21, 2017 at 05:19:28PM +0200, Robert Raszuk wrote:
> enforcement and RFCs like this ?

Yes, and yes.

Even if I cannot discern whether this was meant as rhetoric questions - 
the answer is still yes.  If you assume that 40.000 of those ASes have
noone on-site who has any idea how BGP works, you are likely still too
optimistic about the level of understanding out there.

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 21 Apr 2017 16:45:23 +0100

> optimistic about the level of understanding out there.

+ yes, and yes again.

I am sick to the back teeth of handling the fallout from unintentional
routing leaks.  Incidentally, in my experience, these are NOC size
invariant - large multinational NOCs are just as likely to leak a DFZ as
tiny rural WISPs, which suggests to me that the problems are inherently
tied to non-sensible defaults.

Vendors please note: your defaults cause continual and serious
operational problems.  This is not ok.

Nick

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 17:46:24 +0200

On Fri, Apr 21, 2017 at 05:37:41PM +0200, Gert Doering wrote:
> Even if I cannot discern whether this was meant as rhetoric questions
> - the answer is still yes.  If you assume that 40.000 of those ASes
> have noone on-site who has any idea how BGP works, you are likely
> still too optimistic about the level of understanding out there.

Instead of blaming operators, I prefer we assign some work items to
vendors to motivate them to improve their software rather then acept
this scorn. I'd be careful to indulge Robert in his blatant disrespect
for the operational side of the internet.

So perhaps we can rephrase the rethorical question "Are we really that
bad in Internet NOCs" into "Are NOCs forced to work with poor user
interfaces, inconsistency across implementations, and insecure defaults?"
- Yes, they are. Vendors would do well to take same ownership.

Kind regards,

Job

---

From: Gert Doering <[email-masked]>
Date: Fri, 21 Apr 2017 17:48:36 +0200

Hi,

> - Yes, they are. Vendors would do well to take same ownership.

And this, yes.

(... and I applaud the IOS XR folks for boldly going where no Cisco 
OS has gone before, and demonstrating that "yes, it can be done!")

Gert Doering
        -- NetMaster

---

From: Eric C Rosen <[email-masked]>
Date: Fri, 21 Apr 2017 11:54:29 -0400

Well, it didn't take long for this thread to reach the point of 
diminishing returns.

FWIW, my take on it is the following.

The document should not be considered to be an update to 4271, as it 
does not change anything 4271 says, and does not update or extend the 
protocol in any way.  Standards track seems entirely inappropriate, as 
there is no protocol specification in it.

The document is not really a Best Current Practices document, as it 
advocates a change from current practice.

The document is okay as an Informational document representing the 
consensus of the GROW WG.  It would be nice if the document indicated 
that it only considers the use of BGP when providing Internet service, 
and does not consider other uses of BGP.  It would also be nice if the 
document indicated that there may be unintended side-effects of changing 
the defaults, but the authors consider those to be of no consequence.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 18:00:21 +0200

Gert & Nick

Your answer perfectly proves that what we need is education on how to write
proper BGP policy.

Enforcing by RFC someting which everyone else considers a big secret and
private data is and will be of no solution to the root of the problem.

I am to proceed with this draft fwd if in the same time in the draft itself
we document what proper bgp policy is.

Otherwise it is like enforcing something which those who (in spite of their
good intentions) do not do today will continue not doing tomorrow.

Cheers
R.

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 18:03:07 +0200

On Fri, Apr 21, 2017 at 02:32:01PM +0000, [email-masked] wrote:
> statement document. IMHO the place for this problem statement is in
> the Introduction section of you document.
>  

Your comments are helpful. While I think I understand the intent of the
NEW text I have trouble envisioning how this would work for all vendors,
and as such so far the authors have been consistent in leaving out
implementation guidance and implementation specific procedures. I
believe that each vendor will have its own path towards compliance,
should they choose to conform.

However, through this thread it has become clear some representatives of
some vendors lack the creativity to formulate a path towards compliance,
and we'll take this lack of imagination as a plea for help. I will take
this suggestion under advisement and try to improve upon it.

The risk here is that an already compliant implementation, or an
implementation that intends to become compliant through a major version
bump in the release notes, may feel that the suggested path towards
compliance does not conform to their path, and therefor reject the
suggestion. 

BTW, did you mean to use the draft-farrel-soon-05 definition of "soon"
in the NEW text? ;-)

> - I would also support the addition of a BGP capability to signal
> whether this behavior is enabled or not. This would allow an EBGP peer
> to reject the EBGP session if the configuration is not acceptable to
> them. But I can live without.

Introducing a new capability for this specific seems to impose
additional work on implementations which are already compliant. Perhaps
a more generalised mechanism should be introduced that stands apart from
this draft, one where the BGP speaker signals the version of the
software it is running, or sends a list of all RFCs it is compliant
with. 

> - I would prefer that the BGP session be closed, rather than keeping
> it up and not considering the received routes/not sending routes.

I can see how different people would have different preferences in that
regard, however I'd like to add to the context that a 'session up and
not using received routes / not announcing routes' is a very common
debugging tool in itself. I myself have often times reduced a neighbor's
configuration to the bare minimum to debug issues. If you ask around in
GROW I think you'll find that 'up + reject' is a better starting point
for most operators then 'down'.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 18:19:20 +0200

> Well, it didn't take long for this thread to reach the point of

As is often the case when people engage in polemic debate.

> there is no protocol specification in it.

The Area Director and the IDR Chairs believe that the document actually
should update 4271, and the next version of this document will be an
attempt to codify that. Some of the authors expressed concerns, but we
are never the less willing to put in the work.

Should you be interested in how an update to 4271 could be formulated,
please take a look at:

> advocates a change from current practice.

Eric, you fail to acknowledge that already today, in widely deployed
software, a myriad of behaviours exist in this regard. There is no
"current practice".

> document indicated that there may be unintended side-effects of
> changing the defaults, but the authors consider those to be of no
> consequence.

Your concern is noted.

Kind regards,

Job

---

From: Enke Chen <[email-masked]>
Date: Fri, 21 Apr 2017 09:22:41 -0700

Job,

IMO the most important point from the discussion is that any BGP extension
or behavior change must be backward compatible, which this document is lacking
or even missing.  After more than 20 years of BGP deployment, the world is no
longer "green field" any more.

Regards,  -- Enke

---

From: Eric C Rosen <[email-masked]>
Date: Fri, 21 Apr 2017 12:28:28 -0400

On 4/21/2017 12:19 PM, Job Snijders wrote:
> should update 4271,

Perhaps they will take account of the last call comments.

---

From: Jay Borkenhagen <[email-masked]>
Date: Fri, 21 Apr 2017 12:48:38 -0400

Eric C Rosen writes:
 > [...]  It would also be nice if the 

I am not one of the authors (although for what it's worth I have been
agreeing with their comments in this thread), but I don't think
they're saying those side-effects are of no consequence.

While such side-effects may be unfortunate, they will be felt only by
those who do not read release notes, do not test, do not slow-start
their code upgrades, etc.  That such folks may exist should not
prevent standardizing the sane and important defaults this draft
proposes.

To those who propose keeping the ebgp session down if no policies are
configured, that's worse.  It would require an implementation to tear
down an established ebgp session when a necessary policy is removed,
violating POLA.  The actions proposed in this draft constitute failing
hard (hard enough). 

Thanks.

						Jay B.

---

From: Job Snijders <[email-masked]>
Date: Fri, 21 Apr 2017 22:27:11 +0200

> I personnally would recommend to apply it to only 1/1 and 2/1.

This topic was discussed during the GROW meeting at IETF 97,
subsequently a follow-up action item was defined and executed upon, and
the result passed GROW Last Call

Here is one reference https://www.ietf.org/mail-archive/web/grow/current/msg03718.html

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 21 Apr 2017 22:54:00 +0200

> ive/web/grow/current/msg03718.html

​On the list there was three emails on this including Jon's you quote who
was rather for #1 ...
"IPv4/IPv6 unicast AFI/SAFI EBGP only"

​I do not see any discussion on that which would be anywhere comparable to
number of IDR WG members suggesting to limit it only to 1/1 & 2/1.

Thx,
R.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 23 Apr 2017 14:56:05 +0200 (CEST)

> longer "green field" any more.

I have been involved in running core networks since late 90ties. I've 
deployed several vendors gear. Yes, going from IOS to IOS XR with the 
change to XR having default deny if there is no policy, that was a single 
occasion "oh", and then I knew that. The good part here is that it's 
failsafe "close", so that you don't announce anything by accident. In IOS 
you have to basically paste two lines at once, with the first line being 
the creation of the neighbor, the second line being shutdown. Then you can 
configure the rest. Otherwise there is a race condition in the immediacy 
of a per-line, immediate committing operating system such as IOS.

This is just bad design. It's "fail open" default. If you somehow fail to 
paste that second shutdown line, you're now fully-open, announcing and 
accepting all routes.

If IOS would be changing its defaults, the CLI line migration code could 
by default insert a PERMIT-ALL policy statement, or some other means where 
an upgrade would keep the behaviour of the box intact across operating 
system versions.

So I fully support draft-ietf-grow-bgp-reject-05 because it just makes 
more operational sense than the old default that for instance IOS 
implements. We need default fail-close, because it just creates less 
problems than default fail-open.

If this doesn't make sense, why was it chosen for IOS XR back in the early 
00ds?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 23 Apr 2017 15:42:33 +0200

> If this doesn't make sense,

No one said here that it does not make sense. All comments so far are
rather about few points:

-  what real problem does it solve
-  should it apply to all address families or just 1/1 & 2/1 and all
profiles of BGP use
-  if done should it be silent ? (**)
-  should routes still be in BGP Adj_RIB_In and be sent by BMP (further
hiding the issue)

> why was it chosen for IOS XR back in the early 00ds?

Because 3 first major customers of XR said they prefer AF centric BGP
config and to have such policy for eBGP route announcement. As XR was not
deployed anywhere making that default was zero pain for anyone.

** Making route ineligible for best path and local RIB insertion is one
thing. But if you also are using say "add-paths  all" I would not be
surprised if those bgp implementation which care less if the route is
inserted in RIB and FIB before propagation would advertise it anyway to
iBGP peers (**) making even bigger mess in your network.

- - -

Besides to make sure the BGP policy is there before route is annouced or
accepted very is much simpler way. It is not about changing default, mess
with 100s of corner cases in the code. BGP configuration parser may just
not allow to configure "activate" under a given neighbor in selected (or
all) AFs if policy is not there. Done and makes everyone happy ! On upgrade
to new OS optionally you can still run fine with auto-addition of this
proposed "bgp insecure-mode" global bgp config line.

Example:

router bgp 65000

neighbor  192.168.1.1 remote-as 65001
address-family ipv4 unicast

neighbor 192.168.1.1 activate

                                          ^^^^^^^^^

!!! COMMAND NOT ACCEPTED .. ACTIVATE NOT ALLOWED BEFORE POLICY FOR EBGP
PEER IS SET.

//R

On Sun, Apr 23, 2017 at 2:56 PM, Mikael Abrahamsson <[email-masked]>
wrote:

> configure the rest. Otherwise there is a race condition in the immediacy of
> a per-line, immediate committing operating system such as IOS.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 23 Apr 2017 16:12:49 +0200 (CEST)

> !!! COMMAND NOT ACCEPTED .. ACTIVATE NOT ALLOWED BEFORE POLICY FOR EBGP 

I don't have any problems with this way of solving the issue. The document 
might say either this or to default-deny. Both solve the problem at hand 
(neither is default-open).

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 23 Apr 2017 18:37:30 +0200

Ok so we may have a way forward then ;)

The only issue is such option does not work well with forward referencing
however unlike in XR rest of IOS if I recall does not allow forward
referencing.

So only change AF specific neighbor command sequence in config and we are
done ;).

//R.

---

From: Enke Chen <[email-masked]>
Date: Sun, 23 Apr 2017 13:01:21 -0700

Hi, Mikael:

On 4/23/17 5:56 AM, Mikael Abrahamsson wrote:
> I have been involved in running core networks since late 90ties. I've deployed
> several vendors gear. Yes, going from IOS to IOS XR with the change to XR having
> default deny if there is no policy, that was a single occasion "oh", and then I
> knew that. The good part here is that it's failsafe "close", so that you don't announce
> anything by accident. In IOS you have to basically paste two lines at once, with
> the first line being the creation of the neighbor, the second line being shutdown. 
> Then you can configure the rest. Otherwise there is a race condition in the immediacy

I think the premature session bring up is fixable. Your email has been forwarded to
the IOS development team.

> 
> This is just bad design. It's "fail open" default. If you somehow fail to paste that
> second shutdown line, you're now fully-open, announcing and accepting all routes.
> 
> If IOS would be changing its defaults, the CLI line migration code could by default
> insert a PERMIT-ALL policy statement, or some other means where an upgrade would keep 
the behaviour of the box intact across operating system versions.
> 
> So I fully support draft-ietf-grow-bgp-reject-05 because it just makes more operational
> sense than the old default that for instance IOS implements. We need default fail-close,
> because it just creates less problems than default fail-open.
> 

> If this doesn't make sense, why was it chosen for IOS XR back in the early 00ds?

I don't think that folks are saying this "deny all" doe not make sense.  When you
start a *new* software, certainly you are free to choose a default based on the most
up-to-date knowledge.

As I understand, that was the case for IOS XR as a new and separate software release.
 
Regards,  -- Enke

---

From: Brian Dickson <[email-masked]>
Date: Sun, 23 Apr 2017 21:43:08 -0700

For those not intimately familiar with non-Internet AFI/SAFI use cases (ie
other than 1/1 and 2/1), could someone explain which ASNs get used?

I am wondering if applying the "new rules" only when global ASNs (those not
reserved for private use) would be a suitable alternative to specific
AFI/SAFI combos?

And, to be clear, IMHO, the difference across AFI/SAFI should probably be
MUST vs SHOULD (or maybe MAY).

Also, don't at least some vendors have different code bases depending on
the use case, in which case the default could differ by code base?
E.g. ISP vs Enterprise vs (whatever other groupings/labels exist).

Plus, IMHO, a global knob to change the default is something that would be
wise to implement; I don't know whether recommending that belongs in the
document

Brian

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 24 Apr 2017 08:37:42 +0200

Hi Brian,

Today global ASN is used for any eBGP session irrespective on address
family. There was few requests in the past to implement per AF ASN but to
the best of my knowledge it was never implemented.

MAY for other AFI/SAFIs works for me .. but let observe that if someone
goes with parser based solution this while accomplishes the goal is clearly
not a Standards Track thing but local implementation.

ISP vs Enterprise images differ in features enabled not different behaviour
of the same feature and everybody tries to still build from same branch. So
that distinction would not work.

//R.

On Apr 24, 2017 06:43, "Brian Dickson" <[email-masked]>
wrote:

Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 24 Apr 2017 09:36:13 +0200 (CEST)

On Sun, 23 Apr 2017, Enke Chen wrote:

> I don't think that folks are saying this "deny all" doe not make sense. 
> When you start a *new* software, certainly you are free to choose a 
> default based on the most up-to-date knowledge.

There is no reason a new version of IOS couldn't have changed defaults, 
but CLI migration can fix this to keep the old default by adding a line 
when software is upgraded from an older default-open version.

If the new versio with default-deny has a command that gives syntax error 
in the old IOS, then this can be added to all neighbor configuring 
templates and will work identically on both the new and old version.

---

From: Enke Chen <[email-masked]>
Date: Mon, 24 Apr 2017 11:02:45 -0700

It appears that you have missed a bunch of discussions in the thread.

The CLI migration falls apart when a customer skips releases and jumps
from one release ("permit all") to another ("deny all").

-- Enke

On 4/24/17 12:36 AM, Mikael Abrahamsson wrote:
>> I don't think that folks are saying this "deny all" doe not make sense. When you start a *new*
>> software, certainly you are free to choose a default based on the most up-to-date knowledge.
> 
> There is no reason a new version of IOS couldn't have changed defaults, but CLI migration can
> fix this to keep the old default by adding a line when software is upgraded from an older
> default-open version.
> 
> If the new versio with default-deny has a command that gives syntax error in the old IOS,
> then this can be added to all neighbor configuring templates and will work identically on
> both the new and old version.
>

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Apr 2017 14:09:15 -0400

> from one release ("permit all") to another ("deny all").

Only if they don't read release notes, do not perform any testing, do
not use a staggered deploy, missed the announcements through the NOGs,
missed the customer briefings.

Kind regards,

Job

---

From: Enke Chen <[email-masked]>
Date: Mon, 24 Apr 2017 11:12:37 -0700

> missed the customer briefings.

Unfortunately, yes, especially in the enterprise market ...

Regards,  -- Enke

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 24 Apr 2017 20:19:12 +0200 (CEST)

> from one release ("permit all") to another ("deny all").

I thought this was exactly what the CLI migration code was supposed to do?

Keep track of what versions some default changed in, and fix it when 
migrating the config?

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Apr 2017 14:20:29 -0400

> Unfortunately, yes, especially in the enterprise market ...

You can even tie this to the depreciation cylce, if it takes multiple
years to get there that is fine.

Kind regards,

Job

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 24 Apr 2017 14:55:35 -0400

(catching up from ... not reading a 100 message thread)

> and enforce sequenced upgrade.
isn't this a well known, common practice for software upgrades though?

1) is this software fixing something broken? (or necessary for some feature
we need)
2) does the OS boot/load on a lab/test device (don't laugh, often this
fails!)
3) does the current golden config for this sort of device load on new OS?
4) does expected behavior for device continue to be in effect?
5) are there configuration changes required to get back to the proper
operating state?

I think for a bunch of software upgrades 'make configlet changes' is
required, so ... this doesn't seem out of scope for normal ops work.

> Of course this is all assuming that someone will read carefully the
and test before loading on their whole network of revenue relevant
infrastructure.

> VPN CEs as Bruno mentioned and their provider not following all releases
it's concerning that people have networks with no
filtering/conditioning/etc on their bgp sessions...

>>     >> EBGP use cases and not make it generic for EBGP sessions (defined
>> in
>>     >> 4271).
>>     >>
>>     >> Regards,
>>     >> Keyur
>>     >>
>>     >>
>>     >>> [email-masked] mailing lists by 2017-05-02. Exceptionally,
>> comments
>>     >> may be
>> ballot/

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 24 Apr 2017 14:58:58 -0400

> protocol to be more restrictive, I would expect this to be discussed.

a few times in this discussion people say: "the protocol", but the draft
doesn't propose changing the protocol, it proposes that implementations
stop being wide-open with respect to send/receive conditioning/filtering.

There's no default behavior change in the protocol (which I think doesn't
really talk about filtering prefixes at all, not in the sense of
'prefix-list foo in' anyway).

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 24 Apr 2017 15:01:22 -0400

> without any config changes.

surely people test said code before updating... if they don't then 'there
be dragons' suffices for any upgrade they perform.

---

From: Job Snijders <[email-masked]>
Date: Mon, 24 Apr 2017 15:07:53 -0400

> 'prefix-list foo in' anyway).

Alvaro suggested to update the 4271 text to give clear guidance to
implementors on how this works.

A preview of the forth coming new version of the document is available
here:

We'll be posting a new version tomorrow or so.

Kind regards,

Job

---

From: Jared Mauch <[email-masked]>
Date: Mon, 24 Apr 2017 15:09:18 -0400

> Keep track of what versions some default changed in, and fix it when

	Vendors have never wanted to publish this, nor have they been
willing to truly document it.

	It seems like this is just an excuse to let their whim of the day
be further propogated.

	As someone who actually reads the diffs from the nvgen
ecosystem in platforms like Cisco IOS, it's clear nobody really
reads the full system diffs.
	
	Claiming otherwise is clouding the discussion.  Once again, this is
"We will never ever secure our BGP implementation".  To that, I will
respond accordingly with my account team.

	I also question those who think customers can't easily figure out
how to turn a knob on.  You must really not think highly of your customers.

	- Jared

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 24 Apr 2017 20:59:06 +0100

Enke,

config downgrading is why the "downward-compatible-config" IOS
configuration command exists.  Config upgrading is relatively
straightforward, although it takes years to implement.

Changing defaults and commands is something which has happened on many
occasions over the lifetime of IOS, from radical config breaking "ip
classless", or my personal favourite, "mop enabled" in the interface
config mode, which changed default backwards and forwards on several
occasions, not just once (I'd love to hear the back-story about how and
why that happened, btw).  Other well known examples include the modular
QoS config mechanism which removed the older qos config mechanisms.

Just because a large vendor doesn't like changing defaults, that doesn't
mean it's not a sensible thing to standardise.

No-one is under any illusion that there is a cost to vendors
implementing this but right now, the cost of not doing it is borne by
other people, and we're tired of it.

Nick

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 07:20:42 +0000

Hi Christopher,

Please see inline [Bruno]

From: Christopher Morrow Sent: Monday, April 24, 2017 8:56 PM

On Wed, Apr 19, 2017 at 6:26 PM, Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:
Keyur,

1) is this software fixing something broken? (or necessary for some feature we need)
2) does the OS boot/load on a lab/test device (don't laugh, often this fails!)
3) does the current golden config for this sort of device load on new OS?
4) does expected behavior for device continue to be in effect?
5) are there configuration changes required to get back to the proper operating state?

I think for a bunch of software upgrades 'make configlet changes' is required, so ... this doesn't seem out of scope for normal ops work.
[Bruno] The acceptability may be different, depending on whether the config change is required to enable a feature that you want to enable, or is imposed to you with no benefit in your deployment case.

and test before loading on their whole network of revenue relevant infrastructure.

it's concerning that people have networks with no filtering/conditioning/etc on their bgp sessions...
[Bruno]
- I guess you meant EBGP sessions. Even probably EBGP sessions between administrative boundaries, while there are EBGP sessions within administrative boundaries. This document does not seem to make the latter distinction.
- The context of the comment was about VPN. P stands for private, where the network operator is paid to provide transparent IP network. Filtering is not that transparent, in particular in the middle of the customer network. Typically, the config includes a safeguard policy to limit the number of prefixes accepted from the EBGP session (to protect from overloading routers) but such policy does not seem to count as “import policy” from this draft.

//R

On Thu, Apr 20, 2017 at 12:16 AM, Keyur Patel <[email-masked]<mailto:[email-masked]>> wrote:
And that would be good enough if that would allow exemptions of DC networks and any other networks that may need exemption.

Regards,
Keyur

On 4/19/17, 2:08 PM, "Jared Mauch" <[email-masked]<mailto:[email-masked]>> wrote:

    Jared Mauch

    > On Apr 19, 2017, at 4:58 PM, Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>> wrote:
    >
    > On 4/19/17, 4:43 PM, "Idr on behalf of Keyur Patel" <[email-masked]<mailto:[email-masked]>
    > on behalf of [email-masked]<mailto:[email-masked]>> wrote:
    >
    >> <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:
    >>
    >>   IDR folks,
    >>
    >>> Cc: [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>,
    >> [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 07:25:40 +0000

From: Christopher Morrow Sent: Monday, April 24, 2017 8:59 PM

On Wed, Apr 19, 2017 at 7:30 PM, Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>> wrote:

On 4/19/17, 7:25 PM, "John Scudder" <[email-masked]<mailto:[email-masked]>> wrote:

>On Apr 19, 2017, at 7:18 PM, Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>> wrote:

a few times in this discussion people say: "the protocol", but the draft doesn't propose changing the protocol, it proposes that implementations stop being wide-open with respect to send/receive conditioning/filtering.

There's no default behavior change in the protocol (which I think doesn't really talk about filtering prefixes at all, not in the sense of 'prefix-list foo in' anyway).

[Bruno] There seem to be various opinions about this. This is a bit of a concern at this stage. In particular, some previous comments/support may have used the wrong assumptions.
If this is indeed the goal to make no protocol change, including no default behavior change in the protocol, could the document be updated to state this. In which case, Standard Track may be required.

Thanks,
Regards,
--Bruno

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Apr 2017 09:32:44 +0200 (CEST)

On Tue, 25 Apr 2017, [email-masked] wrote:

> If this is indeed the goal to make no protocol change, including no 
> default behavior change in the protocol, could the document be updated 
> to state this. In which case, Standard Track may be required.

I re-read https://tools.ietf.org/html/draft-ietf-grow-bgp-reject-05

As far as I can see, it doesn't do any protocol change. It changes the 
default behaviour a router comes with when it comes to treating routes 
sent/received when there is no policy map applied on the neighbor.

So it suggest a behavioural change, not a protocol change.

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 07:46:32 +0000

From: DECRAENE Bruno IMT/OLN  Sent: Tuesday, April 25, 2017 9:26 AM

[Bruno2] I meant: in which case, Standard Track may _not_ be required.

Thanks,
Regards,
--Bruno

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 07:54:40 +0000

> From: Mikael Abrahamsson [mailto:[email-masked]]  > Sent: Tuesday, April 25, 2017 9:33 AM

So we now have 3 readings:
- protocol change updating RFC 4271 (base BGP spec)
- a behavioural change, not a protocol change
- no default behavior change in the protocol

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Apr 2017 10:05:59 +0200 (CEST)

> - protocol change updating RFC 4271 (base BGP spec)

Change? As far as I can see, 4271 has nothing on route-policy (just 
reading the TOC). So it seems to me that this would be an addition, not a 
change. Or am I mistaken? So this doesn't seem correct.

> - a behavioural change, not a protocol change

I agree with this.

> - no default behavior change in the protocol

Protocol, no. For me protocol is what goes on-wire, what goes into the 
statemachine, route selection etc.

This is a change in what is the default policy when none is explicitly 
configured. I can't find anything in 4271 that talks about this. So it's 
not a change per se, it's an addition and it's more operational than 
protocol change of behaviour.

So really, this is GROW stuff. It's not necessarily touching core BGP 
protocol documents, unless we absolutely want to.

https://datatracker.ietf.org/wg/grow/about/

"(v). Document the operational aspects of securing the Internet routing 
system, and provide recommendations to other WGs."

So this is exactly what's happening as far as I can tell.

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 08:32:04 +0000

> From: Mikael Abrahamsson [mailto:[email-masked]]  > Sent: Tuesday, April 25, 2017 10:06 AM
> 

Looks good.
But this seems that this would call for an Informational document (vs STD track currently), providing _recommendations_ (i.e. not directly any protocol change) to IDR, and/or vendors, and/or operational community. 

e.g.
1) documenting the operational issues.
My reading would be "Implementations which by default advertise all their BGP routes over an EBGP session plus have a CLI not allowing the EBGP session and its associated policy to be configured atomically create frequent involuntary route leaks in the Internet, including in the absence of any configuration error.
Possibly, this may be only one part of the problem, but here would be a good document to explicit those operational issues. (rather than complying that IDR do not listen to operational feedback)

2) propose recommendations
(e.g. 
- Implementations should support a mode where BGP routes are not advertised unless explicitly required by a route policy.
- As some configurations tasks may require multiple lines, Implementations should support a mode where multiple configurations items are configured atomically
- network operator operating on the Internet routing system should use/enable those 2 features.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Apr 2017 11:42:14 +0200 (CEST)

On Tue, 25 Apr 2017, [email-masked] wrote:

> My reading would be "Implementations which by default advertise all 
> their BGP routes over an EBGP session plus have a CLI not allowing the 
> EBGP session and its associated policy to be configured atomically 
> create frequent involuntary route leaks in the Internet, including in 
> the absence of any configuration error. Possibly, this may be only one 
> part of the problem, but here would be a good document to explicit those 
> operational issues. (rather than complying that IDR do not listen to 
> operational feedback)

https://tools.ietf.org/html/rfc7908 already documents the problem with 
"route leaks". What you're asking for is as far as I can tell already 
documented there.

Perhaps not the *WHY* (as in operator error in configuring things 
correctly), but that's exactly what draft-ietf-grow-bgp-reject is about.

---

From: <[email-masked]>
Date: Tue, 25 Apr 2017 10:09:58 +0000

> From: Mikael Abrahamsson [mailto:[email-masked]]  > Sent: Tuesday, April 25, 2017 11:42 AM
> 

If the problem is route leak (which is indeed a problem to solve):
- draft-ietf-grow-bgp-reject is unfortunately not a solution to route leak. There is nothing to check/unsure that the route advertised/received are the "right" ones.
-  IDR is already working on solutions which seem to have a better coverage. e.g. latest IDR meeting:

     2) Route Leak Prevention using Roles in Update and Open messages [Alexander Azimov/Randy Bush] (10) 
     https://datatracker.ietf.org/doc/draft-ymbk-idr-bgp-open-policy/
     
     3) Route Leak Detection and Mitigation [Sriram] (10)
     https://tools.ietf.org/html/draft-ietf-idr-route-leak-detection-mitigation-06   

So if the problem is route leak, draft-ymbk-idr-bgp-open-policy seems a better solution to me.
Eventually, it could include some idea from draft-ietf-grow-bgp-reject. .e.g.  by introducing a  "semi-strict mode" or "safe mode": if set, and if the "role" capability is not received from the peer, then EBGP speaker MUST behave has per bgp-reject (i.e. not advertising/receiving routes, unless a policy is explicitly configured)

Thanks,
Regards,
--Bruno

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 25 Apr 2017 12:15:34 +0200 (CEST)

On Tue, 25 Apr 2017, [email-masked] wrote:

> - draft-ietf-grow-bgp-reject is unfortunately not a solution to route leak.

Errr... It's a solution to ONE huge glaring reason for some route leaks.

> There is nothing to check/unsure that the route advertised/received are 
> the "right" ones.

The problem draft-ietf-grow-bgp-reject tries to solve is when lack of 
config means you're announcing ALL your routes. So yes, you're technically 
correct, but I don't see the relevance.

> So if the problem is route leak, draft-ymbk-idr-bgp-open-policy seems a better solution to me.

It's not even an WG document. When will it be done? 
draft-ietf-grow-bgp-reject is short, concise and to the point, and easily 
understandable and can be adopted quickly.

> Eventually, it could include some idea from draft-ietf-grow-bgp-reject. 
> .e.g.  by introducing a "semi-strict mode" or "safe mode": if set, and

The whole problem draft-ietf-grow-bgp-reject tries to solve is that people 
are leaking routes WITHOUT SETTING ANYTHING. They're just creating the 
neighbor and then *boom* entire BGP table is sent.

---

From: t.petch <[email-masked]>
Date: Tue, 25 Apr 2017 16:11:39 +0100

----- Original Message -----
From: <[email-masked]>
Cc: "Hares Susan" <[email-masked]>
Sent: Tuesday, April 25, 2017 8:54 AM
> > From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: Tuesday, April 25, 2017 9:33 AM

The current text in RFC 4271 is
"    If the route is learned from an external peer, then the local BGP
      speaker computes the degree of preference based on preconfigured
      policy information.  If the return value indicates the route is
      ineligible, the route .. "

With this I-D, that becomes

"    If the route is learned from an external peer, then the local BGP
      speaker computes the degree of preference based on preconfigured
      policy information.

      If there is no preconfigured policy information, the route is
ineligible.

                                    If the return value indicates the
route is
      ineligible, the route ... "

I think that that is a change that will change what is then transmitted
on the wire so it is a protocol change.  YMMV.

Tom Petch

________________________________________________________________________
_________________________________________________

---

From: Greg Hankins <[email-masked]>
Date: Tue, 25 Apr 2017 16:23:40 -0400

I would like to add a couple of points to this discussion wearing my Nokia
Product Manager hat, and specifically as the person who is responsible
for managing this feature in our routing product line.

Our current BGP defaults in Nokia SR OS are insecure, and we are going to
implement this secure behavior because our customers are asking us for it.
Jared, Job, Mikael, Nick, and others have expressed their demand directly
on the GROW and IDR lists as operators, and we are seeing this same demand
as a router vendor too.

Defaults can be changed, it's not an impossible problem.  We put considerable
thought into changing configuration defaults a long time ago, and have
quite a bit of infrastructure support to make sure that changes are handled
correctly between different software releases during an upgrade or downgrade.
This allows us to change a command syntax or default behavior, and preserve
the intended functionality during various software change scenarios.  First,
we are strict about only introducing changes in major releases, unless for
example there is a security issue that needs to be addressed immediately.
Changing defaults or deprecating a command is a multiyear process that
spans multiple major releases, where we first mark a command as deprecated
and eventually remove it as a configuration option.  Our configuration
infrastructure knows how to translate commands between releases if the
syntax changed, or knows how to handle the cases where a default changes
or a command has been removed.

We also have a clear documentation process for communicating changes
to customers in multiple sections of release notes and documentation.
Each version of the release notes has a detailed section ordered by release
that describes new features, deprecated features, and changed or deprecated
commands.  As people have pointed out, operators may not read the available
information, but this doesn't mean that we don't make careful decisions
to change defaults when necessary.  In addition to the documentation,
we communicate roadmap information during customer interactions such as
EBC visits, planning meetings, training sessions, our customer conferences
and even over social media.

There are a number of scenarios we have considered for changing the insecure
default.  This is not an exhaustive list, but here are some examples:
o ISSU upgrade: old behavior
o ISSU downgrade: old behavior
o Reboot: new behavior
o Device that ships with preloaded software in 2017: old behavior
o Device that ships with preloaded software in the future: new behavior

Lastly, we explicitly added the option to the I-D to have a configuration
knob that configures the insecure behavior:
   o  A BGP speaker MAY provide a configuration option to disable the
      preceding behaviors, but it MUST implement them by default.
So if our customers want the old behavior, then we have a way for
them to enable it.

Greg

-- 
Greg Hankins <[email-masked]>

---

From: Jared Mauch <[email-masked]>
Date: Tue, 25 Apr 2017 17:09:46 -0400

On Tue, Apr 25, 2017 at 12:15:34PM +0200, Mikael Abrahamsson wrote:
> The problem draft-ietf-grow-bgp-reject tries to solve is when lack of config
> means you're announcing ALL your routes. So yes, you're technically correct,
> but I don't see the relevance.
> 

	In private at least one vendor has now conceded this is a bad thing
and is trying to fix it, that's progress.

	I'm looking forward to when BGP starts as secure out of the box
as many other protocols, such as SMTP, etc.

	- Jared

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 25 Apr 2017 17:43:45 -0400

On Fri, Apr 21, 2017 at 11:54:29AM -0400, Eric C Rosen wrote:
> does not change anything 4271 says, and does not update or extend
> the protocol in any way.  Standards track seems entirely
> inappropriate, as there is no protocol specification in it.
> 

FWIW, BCP is probably the best targeted status for the document.  I agree
with you completely that this doesn't document deployed code behavior, but
it is the "best practice" for running a network.

At the end of the day, the document status only impacts conformance suites.
And even those depend on the user saying "test these features".

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 25 Apr 2017 18:11:04 -0400

[I'm picking on this specific message to make my points.]

> Something like the ‘bgp unsafe-ebgp-policy’ could be generated on their respective implementations.  I didn’t think that GROW/IDR needed to tell implementors this level of how to manage their relea…

My own thoughts on this draft haven't really changed since my original
comments at the microphone and in the halls after grow when it was
introduced:

- I'm supportive of this idea.  Safe by default is significantly better than
  unsafe by default.
- Moving to this as a new behavior will be extremely painful.  The word I've
  used internally has been "excruciating".

The tenor of this thread has really taken a disservice by conflating the
"this is a good idea" and the "we're going to make long-standing
implementations non-conformant" via the document status bludgeon.  The
latter isn't the fault of the document authors, but it's where I think we
took a wrong turn.

My personal recommendation is, presuming Alvaro and the IESG can live with
the particular bending of the meaning of BCP, that we go for that status.

This thread has generated a lot of good ideas about how to mitigate the pain
of moving existing implementations to support this practice.  It'd be good
if we summarize those some place.  Perhaps that place should be in the
RFC-to-be.

That said, even with some of the techniques involved, implementors will take
considerable pain in making these changes.  Defaults do change.  Impacted
operators get cranky with them and they cause outages.  Sane developers do
not tend to want to be the one tagged as responsible for the change because
the hate mail in the form of bug reports, regressions, broken tests that comes
back to them. :-)

I'm not the most sane of our developers and I've been spending time since
this was originally proposed in grow to allow such a change in our
implementation and mitigate the results.  The change is easy, the mitigation
is not.  And even I'm sensitive to how much pain this will be.  I dare say
my operator friends can't buy me quite enough beer to make it "easy".

But it's worth continuing the conversation.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 25 Apr 2017 18:18:42 -0400

> Can this document define what he means by "configured export policy".  Possibly by referring to standardized yang models.

Referring to at least the IETF published document:
https://tools.ietf.org/html/draft-ietf-rtgwg-policy-model-01

:   The default policy defined by the model is to reject the route for
:   both import and export policies.

I don't have the openconfig models cached and those are effectively the
working documents, but I doubt this point is likely to have changed.
Perhaps someone doing active work in openconfig can respond to this point.

-- Jeff

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 26 Apr 2017 06:39:25 +0200 (CEST)

> on the wire so it is a protocol change.  YMMV.

If you're blackbox-testing a device with this change, you can't tell if 
it's a changed default or if it's got a policy that denies all. There is 
nothing in the on-wire format that has changed. There is nothing changed 
in how the BGP messages are handled in any way, as opposed to a configured 
default DENY-ALL policy in/out.

So I don't see it as a protocol change. It's a default policy change, but 
not a protocol change.

---

From: t.petch <[email-masked]>
Date: Wed, 26 Apr 2017 10:45:57 +0100

----- Original Message -----
From: "Jeffrey Haas" <[email-masked]>
To: "Jared Mauch" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, April 25, 2017 11:11 PM

> [I'm picking on this specific message to make my points.]

[me too]

> > > You can not set "insecure mode" before you reload the OS as
current OS does not have such knob. Unless you delay the deployment
across N releases and enforce sequenced upgrade.
> >
> > Infact, this is the recommendation that I’ve provided to vendors
that have expressed concerns.  There are many defaults that have not
always been displayed, but things like IOS have “show run all” so you
can see these.
> > Something like the ‘bgp unsafe-ebgp-policy’ could be generated on
their respective implementations.  I didn’t think that GROW/IDR needed
to tell implementors this level of how to manage their release, so this
does seem somewhat out of scope, but a concern I can see needs to be
thought about.
> latter isn't the fault of the document authors, but it's where I think
we

I see it as an operator bludgeon (good word that) to fix the problem
that some experience but without considering the impact thereof.

If you stop propagating routes, then you create black holes.  How many
black holes of what size will this change create?  How will those black
holes be detected?  What will be the damage to the Internet as a result?

I see no sign of this having been considered by those in favour of the
change and would expect this to be in the I-D before now.  Since the I-D
categorises this change as making BGP safer, I am asking how unsafe this
change is making BGP, at the same time.

Tom Petch

---

From: Gert Doering <[email-masked]>
Date: Wed, 26 Apr 2017 11:55:48 +0200

Hi,

> holes be detected?  What will be the damage to the Internet as a result?

Black holes get usually noticed very quickly by those *responsible* for
them.

Routing leaks get noticed by the victims, usually not by those responsible.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 26 Apr 2017 13:28:49 +0200

Hi Gert,

Unless the result is not lost of connectivity but lost of BGP path
redundancy from your AS.

It is quite often and in fact good idea to have dual vendors as your ASBRs.

So unless proper cli is automagically added the problem may only  surface
weeks and months after upgrade and in fact far from given ASBR when
external link from still operational ASBR breaks.

Document should discuss that as well and recommend such auto cli.

//R.

On Apr 26, 2017 5:55 AM, "Gert Doering" <[email-masked]> wrote:

Hi,

Gert Doering
        -- NetMaster

---

From: Jared Mauch <[email-masked]>
Date: Wed, 26 Apr 2017 07:39:54 -0400

> Document should discuss that as well and recommend such auto cli.

	I've only cited CLI as an example of how others have addressed
the concerns raised in the thread.  For the purpose of IDR I don't
wish to mandate that part of a solution, but cite other viable
operational alternatives.

	Saying that a BGP operator needs some "auto-cli" to know how
to do BGP stuff tells me these people should not be trusted with BGP
outside their own labs.

	The pollution here is real, when I was at NTT we worked to
mitigate it via a variety of methods.

	We also had executive escalations that reached my desk where people
didn't do their simple homework of "we had two links, redundancy
didn't work, you took us down big bad $ISP".  

	Once I said "Hi, you are only sending 2/3 of your routes on
your backup session, do you want to fix that now while I observe" they
quickly settled down on their escalation issues.

	Things break, people need to learn how to debug the technologies
they are using.  We should be working to make it simple, yes but not
by foregoing an operator developing their own skills to identify problems
and using their supported vendor paths to triage it.

	Putting text in the draft that it must take the form of
"show bgp neighbor 2001:db8::1 " would favor one vendor over another
and perhaps open the draft up to IPR claims.  I do not welcome these
suggestions.

	The reality is many of the operational leaks we see today are
because of one or multiple failures in the routing ecosystem, be it
a network link, NLRI filter, AS_PATH filter or otherwise.

	- Jared

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 26 Apr 2017 13:40:01 +0200 (CEST)

> Document should discuss that as well and recommend such auto cli.

Let me just understand the reasoning here:

Operator today deploys ASBR without neighbor policy. It sends all routes 
to peer/transit and accepts all routes. This is not noticed by other end 
maximum-prefix and they have filtering in place to drop the "erraneous 
prefixes", so everything is working fine.

Now the router is rebooted with new version of software, that is 
default-deny, and vendor didn't provide CLI parser to migrate config to 
assure consistent behaviour in the new version. After the reboot, all 
routes are denied in/out. Operator doesn't check this now unused link, so 
this state is prolonged for an extended amount of time, all traffic is 
traversing other path, only. Still nobody notices.

Now other link goes down for some reason. Now these ASNs are isolated from 
each other, because there is no backup over transit for some reason, or 
transit becomes congested.

This is the scenario you want analyzed? Or is the scenario you want to 
protect against more non-Internet related, no DFZ amount of routes etc?

Because I find the behaviour I describe above as implausible for any 
decently professionally run network.

---

From: Gert Doering <[email-masked]>
Date: Wed, 26 Apr 2017 13:40:57 +0200

Hi,

> external link from still operational ASBR breaks.

This still hurts those that have no policies configured, which is
how it should be.

"Failing open" hurts totally unrelated networks, which is not what it
should be.

Not hard, isn't it?

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 26 Apr 2017 14:00:45 +0200

Hi Jared,

So let's talk examples ... for my own education.

If you are provider and have 1000 multihomed customers with v4/v6 PI space.

Imagine you want to apply granular policy better then permit all on your
inbound.

How practically (considering that your customers get address block all the
time) you wilk track and adjust policy o  a daily basis ?

You need offline machinery when unless they log in and inform you their
routes will be denied. Moreover your backend must be able to check if
prefix does not belong to someone else .... Oh wait what if someone else
just sold it to them ?

With PA blocks policy works well ... with PI I think not so.

And if you are customer and have 4 prefixes in BGP table thing are fine. If
you by accident become transit and advertise fulm table around I think we
can do better in BGP to protect from it then mandate policy.

Cheers
R.

---

From: Jared Mauch <[email-masked]>
Date: Wed, 26 Apr 2017 08:26:13 -0400

> time) you wilk track and adjust policy o  a daily basis ?

	Proper machinery and automation.

> prefix does not belong to someone else .... Oh wait what if someone else

	Sounds like a great market.  I've seen vendors attempt and
fail at this though.  They also assume vendor lock-in, provider lock-in
and other value additive things.  This is why people build their
own automation and abstraction layers.  This is what any provider with
many network elements to manage is doing today.

	There are still those people who do what I call cut+paste SDN
where they modify existing policy out of a device, from notepad.exe or
their vi buffer.  Emacs users have their own operating system to interact
with first.

> With PA blocks policy works well ... with PI I think not so.

	Works fine if you architect it correctly.

> can do better in BGP to protect from it then mandate policy.

	Actually, providers mandate many things on their customers.
There's many other ways to address the broken behaviors, we have used
prefix-lists, as_path filters, max-prefix in an attempt to stop
the leakage.  It's still only mitigated most of the worst events and
covers up people who are behaving as an insecure BGP speaker.  It's some
mitigation, but the desire of operators is to do better.

	Like I said, at least one vendor has commented in private they
are working on fixing their behavior of their implementations.  This is a
good thing I am hoping will be visible in the measurements of routing leaks
in the coming years.

	- Jared

---

From: Gert Doering <[email-masked]>
Date: Wed, 26 Apr 2017 14:54:17 +0200

Hi,

> With PA blocks policy works well ... with PI I think not so.

We do IRRDB lookups twice a day and build prefix- and as-path filters from 
there.  Filters get uploaded to routers if something changes (with some
safety net for "the tool hickupped, and the result is empty").

As do all my upstreams, as we're not buying from networks that cannot
be bothered to do such basic network hygiene.

I consider this industry best practice, even if some large transit ISPs
consider this "too much work" (which is the other end of the BGP leak
problem).

> can do better in BGP to protect from it then mandate policy.

Evidence shows that, as of today, we can not.

"Shooting" would sound like an alternative and very american way to solve 
the issue, but I'm sure that many IETF members would frown on this.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 26 Apr 2017 15:56:25 +0200

> Evidence shows that, as of today, we can not.

​Have anyone actually tried ?​

The BGP origin validation was at least one attempt.

The other one could be as simple as *"ebgp policy auto"* where based in the
IRRDB and your peer's AS router can build a policy automagically using say
BGPQ3.

http://snar.spb.ru/prog/bgpq3/

Otherwise while Jared, you and perhaps most folks on this list already have
automated ways to build nice and accurate policies I suspect they are those
which do not. And those would either put "allow all" or will now start
looking for hints "what do I put in".

And if the end result is what you are doing twice a day why router's can't
do it themselves assuming IRRDB or any other src of truth is accurate ?

Best,
R.

---

From: Jared Mauch <[email-masked]>
Date: Wed, 26 Apr 2017 10:01:45 -0400

> ​Have anyone actually tried ?​

	Yes.

	Please see the past decade of history.

> do it themselves assuming IRRDB or any other src of truth is accurate ?

	You use the best available data you have as a network operator.

	- Jared

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 26 Apr 2017 10:08:44 -0400

> The BGP origin validation was at least one attempt.
origin validation doesn't protect against 'accidentally i became transit,
whoops!' mistakes.

bgpsec also doesn't protect against this scenario.

There have been a few years worth of papers/analysis out of academia (and
at least 2 drafts in the ietf) talking about the above.

> The other one could be as simple as *"ebgp policy auto"* where based in
> the IRRDB and your peer's AS router can build a policy automagically using
> say BGPQ3.
>
>
are you suggesting that the router build it's filtering directly (on it's
own) from an IRRdb? that seems interesting, but also fraught with peril...

I also see problems in setting up the configuration parts for this, for a
single network it probably isn't rough, but for a more generic solution
it's going to involve more configuration toggling than just enabling a
policy on the peerings. At least you'd need to account for:
  1) which irrdb to pull content from
  2) which protocol to use to do that pulling
  3) authentication?
  4) results qualifications (larger than X, smaller than Y, general content
sanity)
  5) how to match/query the irrdb for the particular peerings on this device
  6) timeouts for operations
  7) scheduling of operations
  8) security bits around the new 'service' enabled on this device

there are other things to account for as well...

> http://snar.spb.ru/prog/bgpq3/
that's very vendor specific :(

> Otherwise while Jared, you and perhaps most folks on this list already
> have automated ways to build nice and accurate policies I suspect they are
> those which do not. And those would either put "allow all" or will now
> start looking for hints "what do I put in".
>
>
vendors who sold gear could probably offer solutions in this space.

> do it themselves assuming IRRDB or any other src of truth is accurate ?
'how often does this data change?'
'how often do I want to refresh peering data on devices (from neighbors)'
'what is the sla for this part of the service'

Some folk do it more 'dynamically' (some providers update 4 or 6 times
day), some folk do it 'less dynamically' (email [email-masked] .. data
updated in 24hrs)

'why not do it more!' has lots of reasons in both directions, but really
that's not the point of this draft anyway.

---

From: Gert Doering <[email-masked]>
Date: Wed, 26 Apr 2017 16:28:37 +0200

Hi,

On Wed, Apr 26, 2017 at 03:56:25PM +0200, Robert Raszuk wrote:
> ???Have anyone actually tried ????
> 

How exactly does "BGP origin validation" protect against people leaking
paths they should not?  All the origin ASes are as valid as they were
originally.

("BGP->IGP->BGP redistribution" is not the most common case - in that
latter case, origin validation plus drop invalid would indeed help)

> do it themselves assuming IRRDB or any other src of truth is accurate ?

The router certainly could.  But that's a much more sophisticated requirement
than "do not announce anything at all, unless explicitely being told to".

(And then, there's the question of "truth", as soon as certain IRRDBs 
are involved)

Gert Doering
        -- NetMaster

---

From: "John G. Scudder" <[email-masked]>
Date: Wed, 26 Apr 2017 11:02:59 -0400

#include individual_contributor_disclaimer

On Apr 26, 2017, at 10:08 AM, Christopher Morrow <[email-masked]> wrote:
> 'why not do it more!' has lots of reasons in both directions, but really that's not the point of this draft anyway.

Seems that way to me, too. 

AFAICT:

- bgp-reject is a case of trying to patch one special case, but an important special case. If the special case is important enough, the cost/benefit analysis works out.
- Perfect is often the enemy of good.
- The case Robert has just raised ("it's difficult for providers to filter their customers") is EXACTLY WHY bgp-reject exists, to put the onus on the customer to configure a filter.
- Putting all our eggs in the basket of "perfect filtering by providers" is known from experience to be imprudent (see also: BCP-38).

$0.02,

--John

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 26 Apr 2017 11:24:27 -0400

> the issue, but I'm sure that many IETF members would frown on this.

Prefix-limit is a fairly standard way to shoot the peering session.  It's a
very clumsy one though.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 26 Apr 2017 11:27:49 -0400

> 'what is the sla for this part of the service'

I will point out that RPKI-rtr (RFC 6810) is effectively a form of this sort
of dynamic update mechanism. Managing the database and its consistency for
the feed it advertises is certainly no different than what is being talked
about, but the signaling of the *results* is certainly not completely out of
scope of existing or prior work.

It's still gross that RtConfig or equivalent still is the common operational
practice.

-- Jeff

---

From: "John G. Scudder" <[email-masked]>
Date: Wed, 26 Apr 2017 12:49:23 -0400

By the way, the general topic of ingress filtering and how to promote it is interesting, but relatively tangential to achieving consensus on what the IDR WG's opinion of advancing draft-ietf-grow-bgp-reject-05 is. It's relevant insofar as it addresses the same broad problem space, but as we delve into details I think it has the potential to become a distraction from the main topic.

If folks want to keep discussing details of how to improve ingress filtering, please consider starting a new thread for that purpose. Thanks!

--John

---

From: Brian Dickson <[email-masked]>
Date: Wed, 26 Apr 2017 11:07:46 -0700

I'll see your $0.02, and raise you $0.02:

The argument against this being only a BCP, has to do with it being,
effectively, a safety issue.

A BCP is fine if the only person injured by making an error is the
operator.
However, this is clearly not the case.
(Or should be clear; PM me if you would like an explanation.)

The analogy would be the difference between general features and safety
features, on automobiles.

General features are a great way for auto manufacturers to differentiate
themselves.
And even SOME safety features can be such a differentiator, if they relate
to occupant protection alone (anti-whiplash headrests, side airbags).
However, ONE feature protects against collateral damage, and is MANDATED as
a result:

     The "brake pedal interlock".

The automobile mandate is both simple, and general: At some point in the
"operate the vehicle" process, the brake pedal must be depressed while
performing some other act, in order to put the vehicle in motion.
(The generality allows different mechanisms to be used on automatic vs
manual gear shifts: automatics require brake pedal to shift out of park;
manuals require brake pedal to start the engine.)
The brake pedal interlock is intended to prevent a number of use cases,
from accidental gear-shift by an adult, to unattended child operation, to
inadequately-trained young adult error.
The result in all of those cases is the same: unintended vehicle
motion/operation, with potential for third party loss of life.

This analogy fits very nicely with the -reject use case: in order to
prevent collateral damage (leaking the entire routing table), regardless of
who or why, the brake pedal (ingress/egress filter) must be applied before
the vehicle (BGP peering session) starts (begins).

For the same reason that government regulations mandate this, rather than
leaving it as a voluntary compliance thing, it behooves us (the standards
setting body for BGP) to make this mandatory.

There's nothing wrong with mandatory things having grandfather clauses
("any vehicle/router sold after such-and-such date"). However, that does
not preclude making the thing mandatory on a going-forward basis.

(Yes, analogies are always imperfect, but the sentiment here fits.)

Brian

On Wed, Apr 26, 2017 at 8:02 AM, John G. Scudder <[email-masked]> wrote:

> - bgp-reject is a case of trying to patch one special case, but an
> important special case. If the special case is important enough, the
> cost/benefit analysis works out.
> - The case Robert has just raised ("it's difficult for providers to filter
> their customers") is EXACTLY WHY bgp-reject exists, to put the onus on the
> customer to configure a filter.
> - Putting all our eggs in the basket of "perfect filtering by providers"
> is known from experience to be imprudent (see also: BCP-38).
>
> $0.02,
>
> --John
>

---

From: Jared Mauch <[email-masked]>
Date: Wed, 26 Apr 2017 14:57:30 -0400

> (Yes, analogies are always imperfect, but the sentiment here fits.)

	Thanks.  This is surely the intent, make BGP safer for
all involved.  The interesting bit here is that some implementions
don't even have the capability to apply the brake pedal prior to
the vehicle moving.  This of course makes it easy to move, but
much harder to stop (and undo the collateral damage).

	- Jared

---

From: Warren Kumari <[email-masked]>
Date: Wed, 26 Apr 2017 15:17:48 -0400

On Wed, Apr 26, 2017 at 10:08 AM, Christopher Morrow
<[email-masked]> wrote:
>> Have anyone actually tried ?
>>

... and I suspect that basically everyone who actually run networks
have done this (and probably more than once.)

My first time was in around 1997 or so - I managed to become transit
(on my Cisco AGS+!) between Global Naps and (IIRC) PSI.

I was turning up Global Naps as a peer, so I log on and type:
router bgp 8120
neighbor 192.0.2.1 peer-as 1784

... and, as I press enter, one of the sysadmin folk turns around and
asks me to hand him a sharpie. While doing so I bump my coffee mug,
spilling 3 week old coffee (and a very cool mold colony I was
culturing) all over my desk. What with the cursing and running to find
paper towels and similar I don't come back to the router for a minute
or two... by which time I can mysteriously no longer reach it. Turns
out that becoming transit between 2 (at the time) large providers over
a T1 makes your router unavailable. Eventually BGP falls down (because
keepalives get starved), and then, before you are able to login again,
it comes up.

Stuff like this happens all the time - bringing peers up without
policy is a: really easy to accidentally do and b: hurts both the
person doing it, and everyone caught in the collateral damage.

> There have been a few years worth of papers/analysis out of academia (and at
> least 2 drafts in the ietf) talking about the above.
>
>>
>> The other one could be as simple as "ebgp policy auto" where based in the
> single network it probably isn't rough, but for a more generic solution it's
> going to involve more configuration toggling than just enabling a policy on
> the peerings. At least you'd need to account for:

   9) making sure that you have announced just enough (and received)
just enough that you can actually reach the irrdb

I'm saddened that something which is basically a trigger safety to
make it harder to shoot myself in the foot is this hard....

W

>> those which do not. And those would either put "allow all" or will now start
> Some folk do it more 'dynamically' (some providers update 4 or 6 times day),
> some folk do it 'less dynamically' (email [email-masked] .. data updated
> in 24hrs)
>

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: Alexander Azimov <[email-masked]>
Date: Thu, 27 Apr 2017 00:52:12 +0300

Hi all,

While I do like idea of making simple fix to avoid leaks of full table
(this how I understand the goal of this document) I'm afraid that this will
only result in change from absence of policy to empty policy. This
requirement of policy existence will not automatically result in
requirement of understanding of its purpose.

Additionally, I would like to argue that full table leaks are still common
now. They do exist, but they are rare, thanks to policies at upstream
providers, or, at least prefix limit.

2017-04-26 22:17 GMT+03:00 Warren Kumari <[email-masked]>:

> >>> > you by accident become transit and advertise fulm table around I
> think
> >>> > we
> it's
> day),

-- 
| Alexander Azimov  | HLL l QRATOR
| tel.: +7 499 241 81 92
| mob.: +7 915 360 08 86
| skype: mitradir
| mailto: [email-masked]
| visit: www.qrator.net

---

From: Joe Provo <[email-masked]>
Date: Wed, 26 Apr 2017 19:44:09 -0400

On Wed, Apr 26, 2017 at 11:02:59AM -0400, John G. Scudder wrote:
> - bgp-reject is a case of trying to patch one special case, but
>   an important special case. If the special case is important enough,
>   the cost/benefit analysis works out.
> - The case Robert has just raised ("it's difficult for providers
>   to filter their customers") is EXACTLY WHY bgp-reject exists, to
>   put the onus on the customer to configure a filter.
> - Putting all our eggs in the basket of "perfect filtering by
>   providers" is known from experience to be imprudent (see also:
>   BCP-38).

This.

My initial thought was why are we still talking about this as 
if it were academic? It was a good idea when the first version 
floated in 2015 and it is an even better idea with two more 
years of operational damage and instability to the global 
Internet.

Won't you think of the packets?  For the low cost of a default
change, we can get ahead of the misery of users when their 
packets are misrouted by fail-open policies. Let's help them
find the best path to their dst. 

-- 
Posted from my personal account - see X-Disclaimer header.
Joe Provo / Gweep / Earthling

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 27 Apr 2017 06:57:12 +0200 (CEST)

> keepalives get starved), and then, before you are able to login again,

This is a perfect example from the real world why a fail-close design is 
needed. What you described, we all have happened to us if we've worked 
long enough in this field.

Trains have brakes that need air pressure to release them. If hose breaks, 
compressor breaks, whatever breaks, the brakes are applied.

A BGP session should never come up and pass prefixes any direction without 
a policy.

Oh btw, another story. I've had happen to me several times that all my 
sessions were flapped causing large outage, because I added or removed 
address family from a peer-group. Why is this still a thing? Why can't we 
dynamically add or remove address families from a running session?

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 27 Apr 2017 06:02:03 -0400

> Why can't we dynamically add or
> remove address families from a
> running session?

On your last point ... has been tried via dynamic capabilities ... Failed
to get properly implemented by multiple vendors ;(

https://tools.ietf.org/html/draft-ietf-idr-dynamic-cap-14

//R.

PS. Btw multisessions attempt falls into same category.

https://tools.ietf.org/html/draft-ietf-idr-bgp-multisession-07

On Apr 27, 2017 12:57 AM, "Mikael Abrahamsson" <[email-masked]> wrote:

This is a perfect example from the real world why a fail-close design is
needed. What you described, we all have happened to us if we've worked long
enough in this field.

---

From: Warren Kumari <[email-masked]>
Date: Thu, 27 Apr 2017 15:26:11 -0400

On Wed, Apr 26, 2017 at 5:52 PM, Alexander Azimov <[email-masked]> wrote:
> Hi all,
>
>
> While I do like idea of making simple fix to avoid leaks of full table (this
> how I understand the goal of this document) I'm afraid that this will only
> result in change from absence of policy to empty policy.

Empty policy is fine -- I have a number of sessions where I
intentionally have no policy. What I don't want to have happen is
*accidental* sessions without policy -- because, Murphy's law says
that that will happen wherever it is most likely to hurt / just as I
get onto a plane / right before a RIPE / NANOG meeting, so all my
closest friends can point and jeer.

> This requirement of
> policy existence will not automatically result in requirement of
> understanding of its purpose.

Yup, 'tis a well known adage that you can't fix stupid. I don't think
of this as forcing people to design perfect (or even sane) policy; I
think that it is more like the covers on electrical outlets. Sure, I
can find a screwdriver and wiggle it about in the socket[0], but the
holes are small enough that I shouldn't accidentally electrocute
myself while plugging in my laptop....

> providers, or, at least prefix limit.

They are rarer these days -- but they do still happen. This doesn't
only stop full table leaks though -- it also stops the stupid "whoops,
I just announced all my networks to provider X, I only meant to
announce this /24, but I didn't get a chance to apply policy yet..."

W

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 27 Apr 2017 15:53:19 -0400

> *accidental* sessions without policy

​BGP peers are usually configured using peer groups or peer templates.
Policy (especially empty one) would be part of such template anyway. So
changing the default ​would not help to prevent anything once the template
is applied. And session without template being applied would in vast
majority of cases not come up anyway.

Only in stub multi-homed small networks which someone configures peers by
hand the new default maybe will help.

As mentioned before for those cases much better would be to have BGP build
in protection.

*Example for new BGP rule: *

*IPv4 and IPv6 eBGP received routes to an AS (routes where AS_PATH contains
other then local ASN) may be advertised over eBGP session to other ASes
only when explicitly allowed by policy. *

Such change in default is actually much more practical to really help in a
lot of cases while not breaking anything to those which have little stub AS
and advertise their PI space of 4 routes. Such change also address cases
where someone just adds once "send all" to template.

Best,
Robert.

---

From: Joe Provo <[email-masked]>
Date: Thu, 27 Apr 2017 18:50:17 -0400

On Thu, Apr 27, 2017 at 03:26:11PM -0400, Warren Kumari wrote:
[snip]
> closest friends can point and jeer.
 
While I don't disagree, and this is INTER-domain Routing WG, I 
feel it would be remiss to not mention the applicability in the
datacenterBGP world. Fail-close drastically simplifies the deploy
and staging scenario, allowing different teams to be loosely
coupled and not require lockstep. 

I also like this in the generally-overlooked area of deprovisioning
for all applications: remove the call to policy and in one whack
the device is no longer propagating to your mesh.

---

From: Randy Bush <[email-masked]>
Date: Fri, 28 Apr 2017 23:13:10 +0900

is there a big enough win to break POLA?

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 28 Apr 2017 10:47:03 -0400

> is there a big enough win to break POLA?

POLA - principle of least resistance ... which I had to ask for a
definition of...

I think Job's (jared pointed out contradictory answers between parts of the
same vendor even!) said this previously: "There is no well known
expectation here", because each vendor does something different :( what's
frustrating as an operator is, sometimes inside a single vendor there is no
consensus.

Setting a well known standard starting point really would be helpful here.

---

From: "John G. Scudder" <[email-masked]>
Date: Fri, 28 Apr 2017 10:49:26 -0400

> is there a big enough win to break POLA?

(Principle of Least Astonishment.)

Yes, that's the main question this thread has been debating, thanks for summarizing. Another way it's been put is to ask whether the cost/benefit analysis holds up. Violation of POLA is on the "cost" side of the ledger. One of the arguments that it should not be considered to be a very high cost is that there are ways to mitigate POLA with careful implementation; some examples have been discussed. I guess you might then add a "designing good software is hard" line item to the "cost" side.

Do you have an opinion as to the answer to your well-summarized question?

--John

---

From: Randy Bush <[email-masked]>
Date: Sat, 29 Apr 2017 08:04:23 +0900

> POLA - principle of least resistance

A is for astonishment

> Setting a well known standard starting point really would be helpful

but the vendors will need to SERIOUSLY warn users affected by possible
change for a number of years.  bill's bait and sushi could go from
release 8 to 42 two years from now.

randy

---

From: Brian Dickson <[email-masked]>
Date: Fri, 28 Apr 2017 17:02:26 -0700

The analogy I would use next, is the US "national electrical code" (itself
a subset of the national fire code).

Specifically, the analogy would be how and when to use "new" NEC code,
which is basically: for new builds, or upgrades that "touch" deployed
infrastructure.
Anything already deployed is de-facto grandfathered, and does not need to
be brought up to code. (But may be by anyone who does not wish to burn to
death.)

And a specific example that fits the use-case of both "fail closed" (or
"open" for electrical circuits), AND the POLA: GFCI outlets.
(GFCI = ground fault circuit interrupt; if the ground connector has any
measurable current, the hot connector has its power turned off within one
"cycle".)

New GFCI outlets (things you wire into an outlet box, and into which
electrical devices get plugged), are REQUIRED to do the following:
- ship in the "tripped" state
- fail to engage unless wired and grounded correctly (engage = hit the
"reset" button to make the outlet electrically "live")

It is now literally impossible to improperly wire a GFCI in such a way that
it provides power without providing GFCI function.

How is does this compare to BGP, under the proposal?

In the specific use case of a new install out-of-the-box:
- configuring BGP globally is the equivalent to wiring an outlet
- configuring a neighbor/address family is the equivalent to hitting the
"reset" button (to try to enable the connection)
- if (and only if) there is a policy on the neighbor, can the BGP session
come up; if (and only if) the outlet is wired correctly can something
plugged in get power

How is this actually POLA?

In the current BGP world, if you configure a BGP neighbor, your router
attempts to bring up a session with the far end. If it succeeds, it
immediately sends/receives/propagates routes whether or not there is a
policy (plus, race condition!).
In the absence of a policy, for a novice engineer, this behavior is
surprising. (It was for me the first time I did it in 1995 (oops), and
others have shared similar experiences.)
Note also, that only when both parties have configured their respective
sessions does peering come up. If the first party forgot to apply policy,
they won't discover the error until the second party brings up their end,
which could be hours or days later.

If the scope of the proposal is limited in a reasonable manner (e.g.
address family, and/or new device, and/or built-in protections to upgraded
code, etc.), no more surprise-by-this-specific-operator-error.

The only "astonishment" should be that vendors (finally) provide code that
stops operators from shooting themselves in the head.
(Shooting the feet is still in the realm of operator error, such as
configured-but-ineffective policy logic.)

Vendors should implement appropriate warnings, of course, but that is a
per-vendor issue, rather than having a sane out-of-the-box behavior which
is consistent within and between vendors.

Brian

---

From: Randy Bush <[email-masked]>
Date: Sat, 29 Apr 2017 09:25:58 +0900

i don't think gfi receptacles are upgraded in place

randy

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 28 Apr 2017 20:37:52 -0400

Actually in US they are ... leave aside that no one else in the world is
using GFIs in the actual outlets ;)

In any case if we are talking GFCI they all have nominal leaking current
which varies again region by region and application ... 6 - 50 mA.

Here we are talking 0 mA !!!

---

From: Randy Bush <[email-masked]>
Date: Sat, 29 Apr 2017 13:30:26 +0900

> Actually in US they are

i will admit to not having installed any recently; the last two were in
march of this year.  but i assure you that back in those ancient pre-iot
times, gfi breakers and receptacles were not field upgradable.  they
could only be replaced wholesale.

but ignoring broken analogies and to the point, in the case of this
draft, the operator is in definite danger of a major oops (that's a
technical term) when field upgrading the software in a router.

despite this exposure, i am mostly of the opinion that the benefits
outweigh the risks with this draft.  but vendors should do their best to
ameliorate the risks with loud warnings.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 29 Apr 2017 05:20:25 -0400

> but vendors should do their best
> to ameliorate the risks with loud
> warnings.

Indeed ...

There is also proposed option to significantly reduce the risk by changing
this default only to protect from becomig an accidental transit I suggested
in this thread already.

It is simple to implement by vendors, does not require any protocol change
and does not affect in any way stub guys which today advertise their PI
prefix out and get default in.

Which btw they must already explicitely enumerate either in "network"
statement or "route/prefix-map" during redistribution. Why to enforce same
thing to be configured multiple times in your config ? That is always error
prone.

Thx
R.