Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

From: "Susan Hares" <[email-masked]>
Date: Fri, 15 Jun 2012 15:03:47 -0400

IDR WG and BGPers: 

While there is a strong debate on draft-uttaro-idr-bgp-persistence-01 as IDR
WG document, I do not see a clear consensus.  We also did not get an
overwhelming number of you to hum either way (11 total). 

Since some of those who voted "no" have suggested alternate text to the
author, I would like to extend the time to consider this draft for 3 more
weeks. 

To accept the draft we need a few more of you to "hum" yes or "hum" no. 

Sue Hares

---

From: "Smith, Donald" <[email-masked]>
Date: Fri, 15 Jun 2012 13:48:37 -0600

Yes. I failed to review this during the last run but would like to.

When packets collide the controllers cease transmission AND wait a random time before retransmission (mostly)!
[email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Susan Hares
> Sent: Friday, June 15, 2012 1:04 PM
> To: [email-masked]
> Cc: 'John G. Scudder'
> Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document -
> 3 more weeks
>
> IDR WG and BGPers:
>
>
>
> While there is a strong debate on draft-uttaro-idr-bgp-persistence-01
> as IDR WG document, I do not see a clear consensus.  We also did not
> get an overwhelming number of you to hum either way (11 total).
>
>
>
> author, I would like to extend the time to consider this draft for 3
> more weeks.
>
>
>

This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly
prohibited and may be unlawful.  If you have received this communication
in error, please immediately notify the sender by reply e-mail and destroy
all copies of the communication and any attachments.

---

From: Randy Bush <[email-masked]>
Date: Sat, 16 Jun 2012 06:56:57 +0900

> as IDR WG document

which is procedurally strange to me.  i said yes to making it a wg
document so we could discuss it.  since we seem to be discussing it
anyway, i guess the hum is really about whether one thinks it should be
a ps, kinda what i thought wglc was, which i don't.  so i guess, no, i
do not support this being a wg document.

randy

---

From: "Gunter Van de Velde (gvandeve)" <[email-masked]>
Date: Mon, 18 Jun 2012 04:14:31 +0200

'hum' ... yes...

G/

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Susan Hares
Sent: vrijdag 15 juni 2012 21:04
To: [email-masked]
Cc: 'John G. Scudder'
Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document -
3more weeks

IDR WG and BGPers: 

While there is a strong debate on draft-uttaro-idr-bgp-persistence-01 as
IDR WG document, I do not see a clear consensus.  We also did not get an
overwhelming number of you to hum either way (11 total). 

Sue Hares

---

From: "Fragassi, Roberto (Roberto)" <[email-masked]>
Date: Mon, 18 Jun 2012 07:49:22 -0500

+1 Support.

With the continued growth of spoke topologies in mobility backhaul, the continued work on this draft as WG has its value for dealing with catastrophic failures.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Friday, June 15, 2012 3:04 PM
To: [email-masked]
Cc: 'John G. Scudder'
Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

IDR WG and BGPers:

While there is a strong debate on draft-uttaro-idr-bgp-persistence-01 as IDR WG document, I do not see a clear consensus.  We also did not get an overwhelming number of you to hum either way (11 total).

Since some of those who voted "no" have suggested alternate text to the author, I would like to extend the time to consider this draft for 3 more weeks.

Sue Hares

---

From: Paul Unbehagen <[email-masked]>
Date: Mon, 18 Jun 2012 08:38:51 -0600

+1

--
Paul

On Jun 18, 2012, at 6:49 AM, Fragassi, Roberto (Roberto) wrote:

> Since some of those who voted “no” have suggested alternate text to the author, I would like to extend the time to consider this draft for 3 more weeks.
>  
> To accept the draft we need a few more of you to “hum” yes or “hum” no.
>  
> Sue Hares
>  
>  
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: "John G. Scudder" <[email-masked]>
Date: Mon, 18 Jun 2012 12:07:33 -0400

On Jun 15, 2012, at 5:56 PM, Randy Bush wrote:

> document so we could discuss it.  

We can discuss anything related to BGP on the IDR list, within reason. A document doesn't need to be chartered to be discussed.

WG adoption means there is consensus for the WG to work on the problem the draft addresses, and to generally start with the approach proposed in the draft, although the latter is subject to discussion.

Or that's how I see it.

> since we seem to be discussing it

Duly noted.

--John

---

From: "Henderickx, Wim (Wim)" <[email-masked]>
Date: Mon, 18 Jun 2012 20:15:07 +0200

+1

From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Unbehagen
Sent: maandag 18 juni 2012 16:39
To: Fragassi, Roberto (Roberto)
Cc: [email-masked]; 'John G. Scudder'; Susan Hares
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

+1

--
Paul

+1 Support.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Friday, June 15, 2012 3:04 PM
To: [email-masked]<mailto:[email-masked]>
Cc: 'John G. Scudder'
Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

IDR WG and BGPers:

Sue Hares

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: Marco Rodrigues <[email-masked]>
Date: Mon, 18 Jun 2012 15:12:07 -0400

Support.

Marco.

From: Susan Hares <[email-masked]<mailto:[email-masked]>>
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: "'John G. Scudder'" <[email-masked]<mailto:[email-masked]>>
Subject: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

IDR WG and BGPers:

Sue Hares

---

From: Randy Bush <[email-masked]>
Date: Tue, 19 Jun 2012 07:19:42 +0900

is there an *operator*, other than at&t, who will speak up for this?

randy

---

From: "Susan Hares" <[email-masked]>
Date: Mon, 18 Jun 2012 18:22:39 -0400

WG:  

I'd like to second Randy's question. If you are an operator that feels this
is a good idea, please send me a note.  Public notes are best.  However, if
you do not want to send a public note - please send a private note.

Sue 
Co-chair

-----Original Message-----
From: Randy Bush [mailto:[email-masked]] 
Sent: Monday, June 18, 2012 6:20 PM
To: Henderickx, Wim (Wim)
Cc: Paul Unbehagen; Fragassi, Roberto (Roberto); [email-masked]; Susan Hares;
'John G. Scudder'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document -
3 more weeks

randy

---

From: "Smith, Donald" <[email-masked]>
Date: Mon, 18 Jun 2012 16:45:52 -0600

I work for Centurylink (Qwest, Embarq, Savvis are all part of CTL now) so I qualify as an operator.
I have implemented BGP Gracefull Restart in the past and am interested in this draft.
But would like a little time to review. I have begun reviewing it but will have to review it with the GR rfc and will probably take me a few reviews before I can say one way or another if I support this beyond my support for a slight extension.

(coffee != sleep) & (!coffee == sleep)
 [email-masked]<mailto:[email-masked]>
________________________________
From: [email-masked] [[email-masked]] On Behalf Of Susan Hares [[email-masked]]
Sent: Monday, June 18, 2012 4:22 PM
To: 'Randy Bush'; 'Henderickx, Wim (Wim)'
Cc: [email-masked]; 'John G. Scudder'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

WG:

Sue
Co-chair

-----Original Message-----
From: Randy Bush [mailto:[email-masked]]
Sent: Monday, June 18, 2012 6:20 PM
To: Henderickx, Wim (Wim)
Cc: Paul Unbehagen; Fragassi, Roberto (Roberto); [email-masked]; Susan Hares;
'John G. Scudder'

randy

________________________________

---

From: Warren Kumari <[email-masked]>
Date: Mon, 18 Jun 2012 14:08:23 -0400

> To accept the draft we need a few more of you to “hum” yes or “hum” no.

I have (re-)read and support adoption.

This is a flavor of kink that I want no part of, but a number of folk feel that this is a useful feature, and who am I to question what they do in the privacy of their own network?

W

-- 
Outside of a dog, a book is your best friend, and inside of a dog, it's too dark to read

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 01:09:11 +0000

How many operators are required? Take a look at the author list. Is there a specific operator who is needed to validate that this document should be adopted?

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Monday, June 18, 2012 6:23 PM
To: 'Randy Bush'; 'Henderickx, Wim (Wim)'
Cc: [email-masked]; 'John G. Scudder'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

WG:  

Sue 
Co-chair

-----Original Message-----
From: Randy Bush [mailto:[email-masked]] 
Sent: Monday, June 18, 2012 6:20 PM
To: Henderickx, Wim (Wim)
Cc: Paul Unbehagen; Fragassi, Roberto (Roberto); [email-masked]; Susan Hares;
'John G. Scudder'

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 19 Jun 2012 11:29:08 +0900

> How many operators are required? Take a look at the author list. Is
> there a specific operator who is needed to validate that this document
> should be adopted?

42

we can all see you have waved your dollars at the vendors.  i seriously
want to know if ops want to deploy this.

essentially, this hack is to deal with a control plane failure when
there is no corresponding data plane failure because the control plane
and data plane are not congruent.  the control plane failure is likely
influenced by its complexity.  i suspect that adding more complexity
will not help a lot.

randy

---

From: Shane Amante <[email-masked]>
Date: Mon, 18 Jun 2012 23:34:06 -0600

I would prefer not to see this adopted as a WG doc.  At a high level, I would rather see a more concentrated focus on implementations providing much more detailed log messages, in particular along the lines of "flight data recorder" type capabilities that would allow operators and vendors to more rapidly isolate the root cause of the issue and then take appropriate action, automatically or manually, to mitigate the issue temporarily or even permanently.  This would be of benefit for not only L2VPN/L3VPN services, (as mentioned in the draft), but also Internet and other applications that use BGP for signaling reachability information.

With respect to this particular draft, I have several concerns, but my primary concern is with the following.

If I'm reading this correctly, this draft states that the DO_NOT_PERSIST community "SHOULD" be disabled by default, which means that (by default) all routes will be set to "persist".  Only through manual config on the part of an SP, will a subset of routes be set to DO_NOT_PERSIST, i.e.: obey the classic/current behavior of BGP, that is be withdrawn when the session fails.  This appears to be a fundamental change to BGP, which I would not want to see deployed "by accident" because this suddenly becomes the default behavior in implementations.  Yes, it sucks, but I would rather see routes withdrawn on session failure (like they are today), rather than hang around until the network is "fixed".  Ultimately, problems of this nature are easy to quickly explain to customer (so the Ops & Vendor folks can concentrate on determining root cause and mitigating it), rather than how long it takes to sort through and try to explain "routing anomalies" that are likely to incrementally build-up the longer routes are held in a STALE state.

Ultimately, I empathize with the co-authors that being in this situation is not fun, nor a good day.  (Been there, done that, got the T-shirts ... happy to say I destroyed them afterward :-).  Furthermore, although it does cost money, which is not easy to come by, I believe that a redundant architecture could be deployed in either the SP's network or by the customer (i.e.: multi-home to two carriers) to largely mitigate these concerns.  Yes, this is not practical in all cases, but it's certainly /possible/ and should be considered recommended before such drastic changes are made to BGP.

Just my own $0.02,

-shane

---

From: "John G. Scudder" <[email-masked]>
Date: Tue, 19 Jun 2012 08:36:35 -0400

Shane,

On Jun 19, 2012, at 1:34 AM, Shane Amante wrote:

> If I'm reading this correctly, this draft states that the DO_NOT_PERSIST community "SHOULD" be disabled by default, which means that (by default) all routes will be set to "persist".

I think this should be understood in the context of "... IF you have enabled the feature in your network." [*] I.e., if you haven't chosen to enable the feature, it would be business as usual, even if your peer sent you the DO_NOT_PERSIST or STALE communities. Seems like the draft should be revised (probably in S. 3) to make this more clear.

I don't know if that changes your opinion any, but FWIW.

--John

[*] Not an actual quote from the draft.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 15:09:56 +0200

Hi John,

You are right reg the "enabling knob part" I send Shane the same comment
few hours back offline ;).

But I would not agree with this comment:

 > I.e., if you haven't chosen
 > to enable the feature, it would be business as usual, even if your
 > peer sent you the DO_NOT_PERSIST or STALE communities.

Today there is no persistence in BGP. At most for the GR timer which is 
relatively short.

Tomorrow when anyone in the co-participating set of domains for a given 
SAFI enables persistance I will be getting prefixes marked as STALE and 
those which are not. That means I can not any longer ignore this marker. 
I must at all of my ingress ASBRs either drop all updates with STALE 
community or lower local pref to make sure those would be only used as 
last resort within my AS.

If I fail to do that I will mix live reachability with flaky one and may 
by normal best path rules prefer the flaky ones for days, weeks or 
months. Worse is that we will have a mix of both in the 
non-participating in the game ASes which will be true nightmare to 
troubleshoot it.

That is a significant change which goes across anyone's single AS who 
wishes to use this technique.

Few more questions:

Did authors of the draft reached clear consensus that persistance does 
not override treat-as-withdraw error handling ? That means that 
regardless of persistance enabled or not withdraws will still be send 
out for the eligible prefixes in the malformed update ?

What happens when router get's fatal malformed attributes or update 
messages which still trigger session down ? Would persistance apply or not ?

In general a bit depending on the answers above what are triggers for 
session down then when persistance applies ?

Thx,
R.

>> If I'm reading this correctly, this draft states that the
>> DO_NOT_PERSIST community "SHOULD" be disabled by default, which
>> means that (by default) all routes will be set to "persist".
>
> I think this should be understood in the context of "... IF you have
> enabled the feature in your network." [*] I.e., if you haven't chosen
> to enable the feature, it would be business as usual, even if your
> peer sent you the DO_NOT_PERSIST or STALE communities. Seems like the
> draft should be revised (probably in S. 3) to make this more clear.
>
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 15:24:56 +0200

Btw ... "business as usual" today on EBGP boundary means to clean all 
communities other then those I do not expect. That means that middle AS 
will drop all STALE/DO_NOT_PERSIST markers and no one will have a way to 
distinguished which prefixes are solid and which are questionable.

One of the authors proposed to add BGP capability to solve this as well 
as legacy ASBR case. Note that for this capability to be effective in 
mixed legacy+upgraded networks it would need to be both for IBGP and EBGP.

Is that the direction this draft is going to take ?

Thx,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 13:52:36 +0000

Shane

                Comments In -Line..

Jim Uttaro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, June 19, 2012 1:34 AM
To: Susan Hares
Cc: [email-masked]; 'John G. Scudder'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

I would prefer not to see this adopted as a WG doc.  At a high level, I would rather see a more concentrated focus on implementations providing much more detailed log messages, in particular along the lines of "flight data recorder" type capabilities that would allow operators and vendors to more rapidly isolate the root cause of the issue and then take appropriate action, automatically or manually, to mitigate the issue temporarily or even permanently.  This would be of benefit for not only L2VPN/L3VPN services, (as mentioned in the draft), but also Internet and other applications that use BGP for signaling reachability information.
[Jim U>] I think you are missing the point if the control plane has melted down then all of our customers are out of service.. This basic premise is unacceptable.. You are speaking of fixing part  and I totally agree that we need more info/telemetry.  This does not change the fact that the customer has had a catastrophic outage...

If I'm reading this correctly, this draft states that the DO_NOT_PERSIST community "SHOULD" be disabled by default, which means that (by default) all routes will be set to "persist".  Only through manual config on the part of an SP, will a subset of routes be set to DO_NOT_PERSIST, i.e.: obey the classic/current behavior of BGP, that is be withdrawn when the session fails.  This appears to be a fundamental change to BGP, which I would not want to see deployed "by accident" because this suddenly becomes the default behavior in implementations.  Yes, it sucks, but I would rather see routes withdrawn on session failure (like they are today), rather than hang around until the network is "fixed".  Ultimately, problems of this nature are easy to quickly explain to customer (so the Ops & Vendor folks can concentrate on determining root cause and mitigating it), rather than how long it takes to sort through and try to explain "routing anomalies" that are likely to incrementally build-up the longer routes are held in a STALE state.
[Jim U>] I will have to bring you along the next time this happens to explain it to the customer..

Ultimately, I empathize with the co-authors that being in this situation is not fun, nor a good day.  (Been there, done that, got the T-shirts ... happy to say I destroyed them afterward :-).  Furthermore, although it does cost money, which is not easy to come by, I believe that a redundant architecture could be deployed in either the SP's network or by the customer (i.e.: multi-home to two carriers) to largely mitigate these concerns.  Yes, this is not practical in all cases, but it's certainly /possible/ and should be considered recommended before such drastic changes are made to BGP.
[Jim U>] I do not understand why we are debating the reqs.

Just my own $0.02,

-shane

Sue Hares

---

From: Shane Amante <[email-masked]>
Date: Tue, 19 Jun 2012 07:57:26 -0600

John,

> I don't know if that changes your opinion any, but FWIW.

Thanks for the clarification.  However, that does not alleviate my concerns on this draft, thus my original opinion still stands.

Thanks,

-shane

---

From: Randy Bush <[email-masked]>
Date: Tue, 19 Jun 2012 23:04:41 +0900

> I think you are missing the point if the control plane has melted down

perhaps a control plane less, not more inclined, to melting should be
considered.

centralization of the control plane seriously erodes congruence of
control and data planes.  this leads to nasty instabilities, as i
believe you have discovered.  adding more complexity to the control
plane is just not gonna fix it.

randy, who likes the reflectors to be in the data path

---

From: "John G. Scudder" <[email-masked]>
Date: Tue, 19 Jun 2012 10:36:24 -0400

> out for the eligible prefixes in the malformed update ?

I do not recall this having been discussed, however given that the draft specifically targets session failure (see S. 4.1) I would expect that if the session has *not* failed (including not failing because of the operation of draft-ietf-idr-error-handling) then persistence wouldn't come into play.

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 14:51:53 +0000

John,

	Comments In-line..

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of John G. Scudder
Sent: Tuesday, June 19, 2012 10:36 AM
To: [email-masked]
Cc: Shane Amante; Susan Hares; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

I do not recall this having been discussed, however given that the draft specifically targets session failure (see S. 4.1) I would expect that if the session has *not* failed (including not failing because of the operation of draft-ietf-idr-error-handling) then persistence wouldn't come into play.
[Jim U>] This is right..  Of course we are assuming that draft-ietf-idr-error-handling is actually agreed to and then built/deployed.. I assume that the behavior in the error handling is configurable and not default behavior. 

--John

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 17:19:33 +0200

I do not understand how the point on centralized RRs is relevant to the 
draft discussed.

BGP control plane can melt by the very same trigger regardless if it is 
centralized and consists of handful of RRs or if RRs are in data plane 
and their number goes in 100s.

One could argue that in the latter case it will take much longer for the 
domain to stabilize.

IMHO the issue is not in the centralized vs distributed control plane 
architecture. The issue is in use of limited number of control plane 
implementations. The more you distribute it the harder it get's to 
deploy more independent control plane brains for the network, hence as 
Jim observed his services suffered.

R.

---

From: Shane Amante <[email-masked]>
Date: Tue, 19 Jun 2012 09:29:14 -0600

Robert,

On Jun 19, 2012, at 7:09 AM, Robert Raszuk wrote:
> Today there is no persistence in BGP. At most for the GR timer which is relatively short.
> 
> Tomorrow when anyone in the co-participating set of domains for a given SAFI enables persistance I will be getting prefixes marked as STALE and those which are not. That means I can not any longer ignore this marker. I must at all of my ingress ASBRs either drop all updates with STALE community or lower local pref to make sure those would be only used as last resort within my AS.
> 
> If I fail to do that I will mix live reachability with flaky one and may by normal best path rules prefer the flaky ones for days, weeks or months. Worse is that we will have a mix of both in the non-participating in the game ASes which will be true nightmare to troubleshoot it.

I also have similar concerns with this draft.  In particular, is there an assumption in here that BGP sessions that are "failed" expected to be held down for the same period of time as the "persist" timer(s)?  Otherwise, wouldn't the BGP sessions automatically try to come back up, indefinitely, leading to more churn of adding/removing the STALE community toward other neighbors each time the "bad" implementation receives a malformed attribute and causes a session to fail?  In some ways, that seems to be making the problem worse in that sessions that may otherwise be providing redundancy are now held down for some lengthy period of time, which the NOC may not be as aware of.  Although the draft tries to cover-off the latter issue by checking for availability to the NEXT_HOP on routes advertised as STALE, that seems to be fighting a symptom of the problem caused by this draft and not addressing the real, underlying root cause IMHO.

In summary, I'd much rather see a more general-purpose solution be developed that is applicable to _ALL_ address families, because that way all services can benefit from it, Ops folks only need to learn/train on one technique/mechanism (regardless of service type), it makes it more straightforward for the IDR WG to enhance/extend BGP down the road w/out (too much) consideration of address-family specific behaviors, etc.

-shane

> That is a significant change which goes across anyone's single AS who wishes to use this technique.
> 
> Few more questions:
> 
> Did authors of the draft reached clear consensus that persistance does not override treat-as-withdraw error handling ? That means that regardless of persistance enabled or not withdraws will still be send out for the eligible prefixes in the malformed update ?
> 
> What happens when router get's fatal malformed attributes or update messages which still trigger session down ? Would persistance apply or not ?
> 
> In general a bit depending on the answers above what are triggers for session down then when persistance applies ?
> 
> Thx,
> R.
> 
> 
>> Shane,
>>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 17:48:46 +0200

Hi Shane,

> In summary, I'd much rather see a more general-purpose solution be
> developed that is applicable to_ALL_  address families, because that
> way all services can benefit from it, Ops folks only need to
> learn/train on one technique/mechanism (regardless of service type),
> it makes it more straightforward for the IDR WG to enhance/extend BGP
> down the road w/out (too much) consideration of address-family
> specific behaviors, etc.

Indeed this is great observation. Moreover the draft on one hand talks 
about session failing while on the other hand authors talk about control 
plane "melting".

Let's make it very clear that if my BGP crashes on PEs even if data 
plane is all fine - this proposal brings nothing to the service 
stability as routes will be removed from RIBs.

GR seems to be solving both cases via it's implicit interaction with RIB 
and FIB. Also in one of the mails Bruno said:

"In short: persistence and GR address largely independent use cases and 
can be enabled independently."

+

"So far, I don't see a reason to mandate GR when persistence is used."

While in the same time draft says when mentioning briefly how to 
synchronize the state:

--

4.3.  BGP session re-establishement

    When a failed persistent BGP session is re-established, the Receiving
    Speaker MUST replace the stale routes by the routing updates received
    from the peer.  Once the End-of-RIB marker for an address family is
    received from the peer, it MUST immediately remove any paths from the
    peer that are still marked as stale for that address family.

--

The similar procedure from RFC4724 section 4.2:

--

    The Receiving Speaker MUST replace the stale routes by the routing
    updates received from the peer.  Once the End-of-RIB marker for an
    address family is received from the peer, it MUST immediately remove
    any routes from the peer that are still marked as stale for that
    address family.

--

To mandate peer that it needs to send EOR marker when session get's 
reestablished you need to agree with your peers on it. Today it is done 
via GR (null) capability.

Moreover if BGP process crashed the above does not apply as RFC4724 
clearly mandates marking the routes in loc-RIB while this draft only 
mentions: "attach the STALE community value"

The difference is that GR stale marker is preserved upon BGP restart 
caused by meltdown of control plane while draft-persistance stale marker 
in the form of BGP community is not preserved.

Two more reasons to use GR as a more general and already deployed solution.

Regards,
R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 16:26:28 +0000

Comments In-Line..

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, June 19, 2012 11:29 AM
To: [email-masked]
Cc: Susan Hares; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Robert,

> If I fail to do that I will mix live reachability with flaky one and may by normal best path rules prefer the flaky ones for days, weeks or months. Worse is that we will have a mix of both in the n…

I also have similar concerns with this draft.  In particular, is there an assumption in here that BGP sessions that are "failed" expected to be held down for the same period of time as the "persist" timer(s)?

[Jim U>] No. The Persist-Timer indicates the length of time that the paths should persist, after this timer expires the paths are withdrawn.   

Otherwise, wouldn't the BGP sessions automatically try to come back up, indefinitely, leading to more churn of adding/removing the STALE community toward other neighbors each time the "bad" implementation receives a malformed attribute and causes a session to fail?  In some ways, that seems to be making the problem worse in that sessions that may otherwise be providing redundancy are now held down for some lengthy period of time, which the NOC may not be as aware of.  Although the draft tries to cover-off the latter issue by checking for availability to the NEXT_HOP on routes advertised as STALE, that seems to be fighting a symptom of the problem caused by this draft and not addressing the real, underlying root cause IMHO.

In summary, I'd much rather see a more general-purpose solution be developed that is applicable to _ALL_ address families, because that way all services can benefit from it, Ops folks only need to learn/train on one technique/mechanism (regardless of service type), it makes it more straightforward for the IDR WG to enhance/extend BGP down the road w/out (too much) consideration of address-family specific behaviors, etc.
[Jim U>] Different AFs have different behaviors. Some are more static than other i.e 3107, L2VPN others more dynamic IPV4.. As made clear in the draft persisting IPV4 AF may not be desirable. Although I have not really looked at it as a use case I have gotten notes describing how it BGP Persistence could be applied.

-shane

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 18:39:24 +0200

> [Jim U>] No. The Persist-Timer indicates the length of time that the
> paths should persist, after this timer expires the paths are
> withdrawn.

The question was different. Isn't it the case that if the session which 
just went down .. comes back in X sec (where X sec is less then your 
persist timer) you will start re-advertising all best paths again 
without the STALE community or start withdrawing previously advertised 
paths if others are best ?

What is the mechanism to stop this churn ?

r.

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 17:23:43 +0000

Hi Robert,

>From: Robert Raszuk
>Sent: Tuesday, June 19, 2012 3:10 PM

[...]

>Today there is no persistence in BGP. At most for the GR timer which is

GR timer is specified to be up to an hour. I would not call this "short", especially with regards to current users' expectations on the network (e.g. 50ms FRR, sub-second convergence)
And that's just the timer for session re-establishment. Then you have to wait for all routes to be learnt on all sessions and then perform best path selection on all routes, before being able to flush the invalid GR stale routes.

Really, if you can handle GR stale routes -which are still preferred and used- for one hour, I'd say you can handle persistence stale routes -which are de-prefered and hence not used if a backup exist- for one hour.

Bottom line, I think the argument: "GR is fine, Persistence is bad" is a bit short.

>last resort within my AS.

That's technically inaccurate. And we have already discussed this.
In short inter-domain persistence and hence the use of the STALE marker is only between ASes which have agreed on using this marker. If you don't believe ISP can achieve such BGP inter domain route filtering, this could probably be automated (e.g. If I have not received a BGP "Persistence" capability from my eBGP peer, then I don't send him STALE routes (i.e. there are withdrawn). IMO, that's a technical detail which can be worked on.

>out for the eligible prefixes in the malformed update ?

As for myself, "yes" for your last question.

>messages which still trigger session down ? Would persistance apply or not ?

IMHO, if BGP session goes down, persistence could apply on the previously received valid route (and again, if GR is used in addition of persistence, GR applies first hence GR already keep the routes as per draft-ietf-idr-bgp-gr-notification-00)

>> [email-masked] https://www.ietf.org/mailman/listinfo/idr

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 17:23:45 +0000

Robert,

>From: Robert >Raszuk
>Sent: Tuesday, June 19, 2012 6:39 PM
>
>

"Persistence" or "no persistence" does not fundamentally change the churn at the NLRI level.
With persistence, NLRI are re-advertised with different path attribute.
Without persistence, NLRI are withdrawn.
In both cases, there are updated and re-advertised.

Agreed, at the BGP update message level, churn is different, depending on NLRI packing. Is this what you meant?

>What is the mechanism to stop this churn ?

GR could help. Provided that you have sufficient faith in the validity of the routes to choose to hide and not propagate the error condition.
And you can use it with or without persistence.

In the GR vs persistence discussion, there is clearly a trade-off between re-advertising the updated route status -which create churn- and not re-advertising it -which will create loss of connectivity if your (more or less) blind assumption turn to be wrong.

I'm fine if you prefer the no churn/GR option. But I have trouble understanding why you refuse to consider that others may favor the other option. Which BTW is more accurate from a routing correctness perspective.

PS: I feel that both of us are looping on this churn point again and again.

---

From: <[email-masked]>
Date: Tue, 19 Jun 2012 17:33:18 +0000

Robert,

>From: Robert Raszuk
>Sent: Tuesday, June 19, 2012 3:25 PM
>
>

For any AS (adjacent, middle...) on the eBGP propagation path:
- either I agreed to use the STALE marker (hence it "expects" the community)
- or the previous BGP speaker will not advertise me the STALE path.

You example does not work.

>mixed legacy+upgraded networks it would need to be both for IBGP and EBGP.

I'm not sure to see what you mean.
On iBGP (hence within the AS), you can assume that the AS can guaranty a consistent behavior. E.g. local_pref are used to lower the preference & on eBGP sessions which have not agreed to use STALE, a BGP policy filter out the route with the STALE community. So there is no issue with legacy.

---

From: "John G. Scudder" <[email-masked]>
Date: Tue, 19 Jun 2012 14:32:47 -0400

> I also have similar concerns with this draft.  In particular, is there an assumption in here that BGP sessions that are "failed" expected to be held down for the same period of time as the "persist…

I don't think so.

> Otherwise, wouldn't the BGP sessions automatically try to come back up, indefinitely, leading to more churn of adding/removing the STALE community toward other neighbors each time the "bad" implementation receives a malformed attribute and causes a session to fail?

Waiting for EoR before running best path and advertising routes should prevent this, though there is scope for excessive local CPU consumption if you get into a lather-rinse-repeat cycle. One way to address this would be to hold down flapping sessions, as the base spec already recommends. The worst pathology would occur if for some reason you had a session that came up, fully converged, *then* went down shortly after that, and then repeated that cycle. It seems fairly unlikely but of course, the universe is large and weird things can and do happen.

The draft is light on discussion of what to do in a flapping situation, this would need to be fixed. Equally, it's light on discussion of whether all session resets should result in persistence, or only certain classes of reset. 

> In some ways, that seems to be making the problem worse in that sessions that may otherwise be providing redundancy are now held down for some lengthy period of time, which the NOC may not be as aware of.  Although the draft tries to cover-off the latter issue by checking for availability to the NEXT_HOP on routes advertised as STALE, that seems to be fighting a symptom of the problem caused by this draft and not addressing the real, underlying root cause IMHO.
> 
> In summary, I'd much rather see a more general-purpose solution

To what problem? I think a big part of the disagreement so far is that folks are not in agreement as to whether the problem statement is worth addressing. The problem statement is kind of buried in S. 1.1:

   BGP Persistence targets the different use case of a catastrophic
   failure when the BGP control plane can remain down for a longer time
   (e.g. hours).

It may be valuable to refocus the discussion on whether folks do, or don't, think the problem statement is worth addressing at all. (I've also pinged GROW on this since they are WGLC'ing draft-ietf-grow-ops-reqs-for-bgp-error-handling-04 right now, and you would think that this topic falls under the heading of "Operational Requirements for Enhanced Error Handling Behaviour in BGP-4".)

> be developed that is applicable to _ALL_ address families, because that way all services can benefit from it, Ops folks only need to learn/train on one technique/mechanism (regardless of service type), it makes it more straightforward for the IDR WG to enhance/extend BGP down the road w/out (too much) consideration of address-family specific behaviors, etc.

I couldn't agree more that all things being equal, orthogonal and general solutions are to be preferred. 

--John

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 20:40:10 +0200

Hi Bruno,

>> Btw ... "business as usual" today on EBGP boundary means to clean
>> all communities other then those I do not expect. That means that
>> middle AS will drop all STALE/DO_NOT_PERSIST markers and no one
>> will have a way to distinguished which prefixes are solid and which
>> are questionable.
>
> For any AS (adjacent, middle...) on the eBGP propagation path: -
> either I agreed to use the STALE marker (hence it "expects" the
> community) - or the previous BGP speaker will not advertise me the
> STALE path.
>

We are discussing current draft not it's future revisions. In the 
current draft there is no EBGP capabilities defined ... my example 
applies fully.

>> One of the authors proposed to add BGP capability to solve this as
>> well as legacy ASBR case. Note that for this capability to be
>> effective in mixed legacy+upgraded networks it would need to be
>> both for IBGP and EBGP.
>
> I'm not sure to see what you mean. On iBGP (hence within the AS), you
> can assume that the AS can guaranty a consistent behavior. E.g.
> local_pref are used to lower the preference & on eBGP sessions which
> have not agreed to use STALE, a BGP policy filter out the route with
> the STALE community. So there is no issue with legacy.

Of course there is. As I have provided the description in the former 
email persistence relies on End-of-RIB to sync it's state when the 
session comes back up. Without IBGP capability you have no way of 
indicating that to your peer.

--

Bottom line: Adding BGP capability negotiation and defining it's format 
to overlap or not overlap with GR is not a "minor detail". It 
fundamentally changing the operational model of the proposal.

For me personally this is like supporting the draft or maybe better 
said: "not being against it" or not.

Best,
R.

---

From: Enke Chen <[email-masked]>
Date: Tue, 19 Jun 2012 12:24:36 -0700

Hi, John:

Thanks for pointing out the importance of the "problem statement" in 
this discussion.  Yes, this topic does fall into the category of 
error/fault recovery, and we will need to evaluate and strike the 
balance between simplicity vs complexity, and common cases vs rare 
cases, etc al.

IMO the complete failure of the redundant control plane falls into the 
category of "rare cases".  They should be addressed from the "root 
cause" instead of building more bandages and complexities both 
operationally and in the software.

-- Enke

On 6/19/12 11:32 AM, John G. Scudder wrote:

[snip]

---

From: Shane Amante <[email-masked]>
Date: Tue, 19 Jun 2012 13:28:38 -0600

Hi John, Enke,

On Jun 19, 2012, at 1:24 PM, Enke Chen wrote:
> Hi, John:
> 
> Thanks for pointing out the importance of the "problem statement" in this discussion.  Yes, this topic does fall into the category of error/fault recovery, and we will need to evaluate and strike the balance between simplicity vs complexity, and common cases vs rare cases, etc al.
> 
> IMO the complete failure of the redundant control plane falls into the category of "rare cases".  They should be addressed from the "root cause" instead of building more bandages and complexities both operationally and in the software.

I completely agree with Enke.

-shane

---

From: "Smith, Donald" <[email-masked]>
Date: Tue, 19 Jun 2012 13:52:27 -0600

Why isn't this:
2.1.  DO_NOT_PERSIST

   This memo defines a new BGP community, DO_NOT_PERSIST, with value TBD
   (to be assigned by IANA).  Attaching of the DO_NOT_PERSIST community
   SHOULD be controlled by configuration.  The functionality SHOULD
   default to being disabled.

This:
2.1.  PERSIST

   This memo defines a new BGP community, PERSIST, with value TBD
   (to be assigned by IANA).  Attaching of the PERSIST community
   SHOULD be controlled by configuration.  The functionality SHOULD
   default to being disabled.

I don't think persist should be the default and dislike negative variables (do_not_....) as it makes following the logic a bit more difficult.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 19:52:19 +0000

All,

	This is a real problem with real consequences. It is not as rare as you think and when it occurs it is a total outage. This is simply not acceptable for a protocol who functionality serves many use cases today. The issue with addressing the "root cause" is that there are many "root causes". I am tired of discovering them by seeing the network fail. We can preserve the network forwarding state in the face of this type of failure and fix the "root causes" without a catastrophic outage..

I am concerned that folks simply do not get it.. The network is only relevant in that it is there to support customers, it does not exist for any other purpose. It needs to be made as reliable and resilient as possible for all BGP use cases. 

The BGP Persistence draft is a simple straightforward approach to maintaining the network in the face of catastrophic control plane failure. We are not dictating it's use, don't use it if you do not want to, but don't state/imply/infer etc that this is not a real issue. It is.

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, June 19, 2012 3:29 PM
To: Enke Chen
Cc: Susan Hares; [email-masked]; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Hi John, Enke,

-shane

---

From: "Smith, Donald" <[email-masked]>
Date: Tue, 19 Jun 2012 13:55:30 -0600

Maybe it is my google fu failing me but several times I have gone looking for a drawing of the BGP +GR FSM and haven't found one. I have seen many drawings of the BGP FSM but none with GR and its timers. Anyone have one or know of one?

Ignorance is Bliss. "Bliss (Basic Language for Implementation of System Software) was a
systems programming language originally for the PDP-10 and DECsystem-20 written at CMU." Kevin Oberman RTD
[email-masked]

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 19 Jun 2012 20:02:22 +0000

Donald,

	We selected the negative as it seemed logical that a session that had been configured for Persist should then assume that the exception is DO_NOT_PERSIST.. See section 3 below..

" 3.  Configuration (Persistence Timer and DO_NOT_PERSIST Community)

   Persistence is configured on a per session and per AFI/SAFI basis.
   Through the use of an inbound BGP policy selectively setting the
   DO_NOT_PERSIST community, the persistence behavior can be set on a
   per route basis.  A speaker configures the ability to persist
   independently of its peer.  There is no negotiation between the
   peers.  A timer must be configured indicating the time to persist
   stale state from a peer where the session is no longer viable.  This
   timer is designated as the persist-timer.  A speaker may also attach
   the DO_NOT_PERSIST community value indicating if a path to a route
   should not persist."

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Smith, Donald
Sent: Tuesday, June 19, 2012 3:52 PM
To: [email-masked]'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Why isn't this:
2.1.  DO_NOT_PERSIST

This:
2.1.  PERSIST

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Jun 2012 22:26:28 +0200

> It needs to be made as reliable and resilient as possible for all BGP use cases.

True.

But it needs to be engineered correctly to work well. You are proposing 
a bandage to cover for bad control plane architecture. This is fine if 
this is just within your network boundaries. Unfortunately draft -01 
goes beyond that at this point.

IMO the right design would be to provide robust x86 based control plane. 
I admit we do not have today a ready commercial way to build one as it 
needs to be seamless to clients (I am not talking about more then two 
IBGP sessions) and really-multi-vendor which goes against any vendor's 
business case. Perhaps those who have in-house multiple BGP 
implementations could provide one ....

But operators are free to enhance what's commercially available with 
their own development. No need to modify any shipping BGP implementation 
or make any changes to BGP protocol is needed.

Best regards,
R.

---

From: Randy Bush <[email-masked]>
Date: Wed, 20 Jun 2012 07:31:24 +0900

> To what problem? I think a big part of the disagreement so far is that
> folks are not in agreement as to whether the problem statement is
> worth addressing. The problem statement is kind of buried in S. 1.1:
> 

why keep the bgp control plane up when you can not deliver packets?  oh,
you mean the control plane is not congruent with the data plane?  then i
think you should say so.

randy

---

From: Randy Bush <[email-masked]>
Date: Wed, 20 Jun 2012 07:32:05 +0900

+1

---

From: Randy Bush <[email-masked]>
Date: Wed, 20 Jun 2012 07:37:48 +0900

jim,

> This is a real problem with real consequences. It is not as rare as
> you think and when it occurs it is a total outage.

why is it not being seen in a lot of other folk's networks?

> The issue with addressing the "root cause" is that there are many
> "root causes".

in your case, i suspect there are a few key ones.  but this may not be
the place to discuss them.

> I am tired of discovering them by seeing the network fail.

then fix it, don't make it even more complex.

randy

---

From: <[email-masked]>
Date: Wed, 20 Jun 2012 07:34:55 +0000

Hi Robert,

>From: Robert Raszuk [mailto:[email-masked]]
>Sent: Tuesday, June 19, 2012 8:40 PM
>
>Hi Bruno,
>

No. Agreement between ASes do not require BGP capability. E.g. People can talk.

>indicating that to your peer.

No.
1) It has been considered that End-of-RIB is a widely available feature
2) Persistence does not rely on End-of-RIB. You can use a timer to fix an upper bound on the convergence (e.g. cf Selection_Deferral_Timer in GR). Then End-of-RIB is simply an add-on to speed up to convergence. Then if you really feel speeding a the convergence is needed (could be discussed how much minutes matter after a 1 hour BGP freeze), all you need is ask for EoR support on your router. No need for capability.
3) Finally, if you ever needed to know the EoR capability of your peer, there is already a capability for this (cf GR).

Capability negotiation is not required.
Even if it were, IMHO I feel that this is a small point compared to the BGP persistence proposition, which could be discussed & challenged on bigger aspects. However, I surely can be wrong and you have a bigger experience than me on designing protocols. 

>said: "not being against it" or not.

Ack.

Regards,
Bruno

---

From: <[email-masked]>
Date: Wed, 20 Jun 2012 08:01:40 +0000

Enke,

>From: Enke Chen
>Sent: Tuesday, June 19, 2012 9:25 PM
>
>Hi, John:
>
>this discussion. 

+1

> Yes, this topic does fall into the category of
>error/fault recovery

+1

> and we will need to evaluate and strike the

True, but the statement is a bit too general so far. The hard point is the evaluation, not just stating that it's needed
Independently of the persistence draft, I would be interested in such evaluation " evaluate and strike the balance between simplicity vs complexity, and common cases vs rare cases" for different features. E.g. BGP persistence, BGP Non Stop Routing, BGPSEC, multiple software train ...

>category of "rare cases".  

Agreed.

>They should be addressed from the "root

IMO this is a good proposition to think beyond the BGP persistence proposition. Thanks to John and you for this. And I think I would welcome and consider better propositions.

And I agree that fixing the root causes are required and better than using BGP persistence (IMHO it's self evident that we all prefer not having failures to deal with).

Now, do you mean fixing the root causes before or after the failures?
If it's after, I feel that this does not address the concern of people experiencing the network issues.
If it's before, IMHO this may not be so easy as the causes may be diverse and hard to identify upfront. Clearly some are SP specific, others vendors specific, others some combination of sub-optimalities on both sides.
Since you are proposing to address the root causes, IMO, at the minimum you need to tackle the vendor side. E.g. what do you propose to improve the situation? When? what are the proposed guaranties or metrics to measure the improvement? And BTW, this should not be limited to BGP. Some points are vendor specific and I'm ready to have an offline discussion and meeting on this. The point is that you can't disregard the persistence proposal to say that it's better to work on the root causes, without working and solving theses root causes.

Thanks,
Regards,
Bruno

>> To what problem? I think a big part of the disagreement so far is that folks
>are not in agreement as to whether the problem statement is worth addressing.
>The problem statement is kind of buried in S. 1.1:
>>
>> It may be valuable to refocus the discussion on whether folks do, or don't,
>think the problem statement is worth addressing at all. (I've also pinged GROW
>on this since they are WGLC'ing draft-ietf-grow-ops-reqs-for-bgp-error-
>handling-04 right now, and you would think that this topic falls under the
>heading of "Operational Requirements for Enhanced Error Handling Behaviour in
>BGP-4".)
>>
>>
>

---

From: <[email-masked]>
Date: Wed, 20 Jun 2012 08:22:44 +0000

Robert,

>From: Robert Raszuk
>Sent: Tuesday, June 19, 2012 10:26 PM
>
>business case. 

Idem, thanks for the proposition and for thinking beyond the persistence draft.
IMHO that may be a valid thing to try, at least on the RR side of the BGP session. Yet, IMHO building a good (reliable, performant) BGP implementation is hard.

However, that would only cover the (some) software issues on the RR side. I see persistence as a bigger safety net.

> Perhaps those who have in-house multiple BGP implementations could provide one ....
>their own development. 

I don't think operators would do a better job on the BGP implementation side. Building software and operating a network is a different business and requires different skills. An open source effort may have value to openly and collectively identify and correct bugs but this is a much wider discussion. And a very long term effort.

Best regards,
Bruno

>No need to modify any shipping BGP implementation

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 20 Jun 2012 10:31:29 +0200

> Yet, IMHO building a good (reliable, performant) BGP implementation is hard.

True. And changing it's fundamental behaviour every few months does not 
help to make it reliable/ performant either ;)

> I don't think operators would do a better job on the BGP implementation side.

I am not asking for that at all. I am asking to simply use multiple 
implementations in the backend. Not two .. but 4 or 5. Probability that 
all fail/melt with the same bug I think is very very low.

Of course I admit that if you one uses service which only single vendor 
supports in BGP then one get's a bit stuck with the risk. And that seems 
to be one of the silent "problem statement" here.

Rgs,
R.

---

From: Warren Kumari <[email-masked]>
Date: Wed, 20 Jun 2012 09:34:48 +0100

> I have (re-)read and support adoption.

When I read this I understood that this would cause less, not more churn (figuring that a failure within an AS this would be less likely to result in withdrawl / readvertisement), but it seems I didn't have my brain fully engaged...

I was also viewing this with a "standard" design in place and so only kicking in in rare cases (such as when one device ups and dies and the other device *in the forwarding path* loses its RE / CP) and didn't consider how often this would occur in other designs. 

Having read the thread, having a better understanding of the draft and having considered it further I withdraw my support…

> This is a flavor of kink that I want no part of, but a number of folk feel that this is a useful feature, and who am I to question what they do in the privacy of their own network?

I stand by my "your network, your rules" stance, but it seems that this won't actually be in the privacy of your own network…

W

--
It's a mistake trying to cheer up camels. You might as well drop meringues into a black hole. -- Terry Prachett

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 20 Jun 2012 14:42:38 -0400

+1

Regards,
Jeff

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 21 Jun 2012 17:58:45 -0400

Randy:

[co-chair hat off]
Your point is well-taken.  
Data planes can continue while BGP drops, but I do not understand the point
of doing heroic efforts to keep the BGP control plane up when the data plane
is down.  Perhaps someone can enlighten me. 

[co-chair hat on]

Sue Hares 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Randy
Bush
Sent: Tuesday, June 19, 2012 6:31 PM
To: John G. Scudder
Cc: idr wg
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document -
3 more weeks

>    failure when the BGP control plane can remain down for a longer time

why keep the bgp control plane up when you can not deliver packets?  oh, you
mean the control plane is not congruent with the data plane?  then i think
you should say so.

randy

---

From: "Susan Hares" <[email-masked]>
Date: Wed, 20 Jun 2012 13:25:11 -0400

Shane:

Thank you for your comments on the concerns from an operator's viewpoint. 

Sue 

From: Shane Amante [mailto:[email-masked]] 
Sent: Tuesday, June 19, 2012 1:34 AM
To: Susan Hares
Cc: [email-masked]; 'John G. Scudder'
Subject: Spam:********, Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR
WG document - 3 more weeks

I would prefer not to see this adopted as a WG doc.  At a high level, I
would rather see a more concentrated focus on implementations providing much
more detailed log messages, in particular along the lines of "flight data
recorder" type capabilities that would allow operators and vendors to more
rapidly isolate the root cause of the issue and then take appropriate
action, automatically or manually, to mitigate the issue temporarily or even
permanently.  This would be of benefit for not only L2VPN/L3VPN services,
(as mentioned in the draft), but also Internet and other applications that
use BGP for signaling reachability information.

With respect to this particular draft, I have several concerns, but my
primary concern is with the following.

If I'm reading this correctly, this draft states that the DO_NOT_PERSIST
community "SHOULD" be disabled by default, which means that (by default) all
routes will be set to "persist".  Only through manual config on the part of
an SP, will a subset of routes be set to DO_NOT_PERSIST, i.e.: obey the
classic/current behavior of BGP, that is be withdrawn when the session
fails.  This appears to be a fundamental change to BGP, which I would not
want to see deployed "by accident" because this suddenly becomes the default
behavior in implementations.  Yes, it sucks, but I would rather see routes
withdrawn on session failure (like they are today), rather than hang around
until the network is "fixed".  Ultimately, problems of this nature are easy
to quickly explain to customer (so the Ops & Vendor folks can concentrate on
determining root cause and mitigating it), rather than how long it takes to
sort through and try to explain "routing anomalies" that are likely to
incrementally build-up the longer routes are held in a STALE state.

Ultimately, I empathize with the co-authors that being in this situation is
not fun, nor a good day.  (Been there, done that, got the T-shirts ... happy
to say I destroyed them afterward :-).  Furthermore, although it does cost
money, which is not easy to come by, I believe that a redundant architecture
could be deployed in either the SP's network or by the customer (i.e.:
multi-home to two carriers) to largely mitigate these concerns.  Yes, this
is not practical in all cases, but it's certainly /possible/ and should be
considered recommended before such drastic changes are made to BGP.

Just my own $0.02,

-shane

IDR WG and BGPers:

Sue Hares

---

From: "Senad .Palislamovic" <[email-masked]>
Date: Fri, 22 Jun 2012 07:58:20 -0400

Apologize for the delay on comments,

Robert,

After going through all the emails, I admit, you've made a lot of valuable
comments that 'seriously' MUST be considered in the new version of the
draft.  However, putting that aside, I would really expect this to be saved
for WG discussions.  I would think that at this stage, we are more less
agreeing to take a look at this problem and its solution space.

Robert, Randy, Shane,

The main question I have for you is why object something that might not
even affect you.  Draft's primary purpose is for the protection of the
control plane for non-internet services (vpls, l2vpn, or infrustructure
routes, i.e. BGP-3107), which in all cases does not affect you or me in any
shape or form.  For folks to chose to extend this to l3vpn or even to
IPv4/IPv6 space, they are doing it consciously knowing the nature of their
network and its dynamics.  If you do see a route with the STALE community,
and do not desire to use this path due its "unreliable" nature, you can
request your provider to withdraw the routes with STALE community
effectively making them DO_NOT_PERSIST at the source which effectively
mimics the standard GR behavior; as Bruno stated, at eBGP, people are
talking and negotiating.

So given all that, help me understand how could this impact anyone who
chooses not to use it.  Am I missing anything?

On the side note, Susan, I do support this draft as WG document.

Senad

On Wed, Jun 20, 2012 at 4:31 AM, Robert Raszuk <[email-masked]> wrote:

> implementations in the backend. Not two .. but 4 or 5. Probability that all
> fail/melt with the same bug I think is very very low.
>
> supports in BGP then one get's a bit stuck with the risk. And that seems to
> be one of the silent "problem statement" here.
>
> Rgs,
>
> R.
>
> ______________________________**_________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/idr<https://www.ietf.org/mailman/listinfo/idr>
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 12:20:40 +0000

Sue,

	I think there is some confusion.. The BGP Control is down not up, while the data plane is up not down. Unlike the internet there are many services that do not have the control and data planes symmetric although even in these cases where there is tunneling I am not so sure it matters. . 

The situation is if we lose the control plane we want forwarding state to persist as long as the NH of the path is still valid. The path should be considered as path of least resort..

There as services such as 3107 ( Labeled BGP ), L2VPN, etc... which use BGP for signaling and dynamic discovery.. SO let's look at L2VPN, BGP is used to dynamically discover and signal state to create PWs which are then used as part of a customer's VPLS ethernet network. There is no exchange of routing state outside of creating PWs internal to the operator or between cooperating operators. BGP is used here to create a static set of PWs that can only be formed in response to config changes on a PE.. So if we lose the signaling path i.e RR, this in no way implies that the forwarding state is suspect.

We need to consider all of the services that are using BGP. As stated in the draft the use case is not directed to IPV4 or the Internet, it would be great if I could hear any comment about the AFs and services that draft is directed towards.

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Thursday, June 21, 2012 5:59 PM
To: 'Randy Bush'; 'John G. Scudder'
Cc: 'idr wg'
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Randy:

[co-chair hat off]

[co-chair hat on]

Sue Hares 

-----Original Message-----

randy

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 22 Jun 2012 15:07:03 +0200

Hello Senad,

Many thx for your comments.

As you see from my original post I asked to limit the scope of the draft 
only to services AFI/SAFIs (eg VPLS). The authors answer was NO .. as 
they may transport their infrastructure routes in Internet SAFI and they 
want to make them persistent.

To your point .. marking the route with a community is a very fragile 
marker. Imagine I am peering to IX which in turn is connected to 500 
peers. I can not imagine how I can "talk with people" to make sure I 
will not get any STALE routes or that my DO_NOT_PERSIST marked routes 
will not be cleaned from that community two ASes down the road. That 
simply does not work across more then two directly connected ASes for 
any AFI/SAFI (and 3107 may easily extend over more then two ASes even 
today).

At most I may enforce to negotiate BGP capability so when I am not 
configured for persistence and not aware of it my peer will send me 
withdraw. That is not "talk-talk" thing, but must be hard assured in the 
proposal by use of real BGP capabilities.

BGP state synchronization when the session comes up using EOR requires 
GR capability which authors claim is not mandatory for this proposal.

Last .. the session down event while BGP process is still healthy and 
running is being addressed by various other proposals in IDR and GROW 
which do target opposite solution .. keep the session up even if some 
errors which would trigger it down are received. So this draft going 
against that will generate a lot of confusion in the field which 
solution to use if IETF folks are going in many directions between 
themselves. I am not making any judgement here which way is better .. 
just observing the proposals.

Regards,
R.

> After going through all the emails, I admit, you've made a lot of
> valuable comments that 'seriously' MUST be considered in the new version
> of the draft.  However, putting that aside, I would really expect this
> to be saved for WG discussions.  I would think that at this stage, we
> are more less agreeing to take a look at this problem and its solution
> space.
>
> Robert, Randy, Shane,
>
> routes, i.e. BGP-3107), which in all cases does not affect you or me in
> any shape or form.  For folks to chose to extend this to l3vpn or even
> to IPv4/IPv6 space, they are doing it consciously knowing the nature of
> their network and its dynamics.  If you do see a route with the STALE
> community, and do not desire to use this path due its "unreliable"
> nature, you can request your provider to withdraw the routes with STALE
> community effectively making them DO_NOT_PERSIST at the source which
> effectively mimics the standard GR behavior; as Bruno stated, at eBGP,
> people are talking and negotiating.
>
> On Wed, Jun 20, 2012 at 4:31 AM, Robert Raszuk <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>     True. And changing it's fundamental behaviour every few months does
>     not help to make it reliable/ performant either ;)
>
>
>     implementations in the backend. Not two .. but 4 or 5. Probability
>     that all fail/melt with the same bug I think is very very low.
>
>     Of course I admit that if you one uses service which only single
>     vendor supports in BGP then one get's a bit stuck with the risk. And
>     that seems to be one of the silent "problem statement" here.
>
>     Rgs,
>
>     R.
>
>     _________________________________________________
>     Idr mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/__listinfo/idr
>     <https://www.ietf.org/mailman/listinfo/idr>
>
>
>
>

---

From: "Senad .Palislamovic" <[email-masked]>
Date: Fri, 22 Jun 2012 10:24:09 -0400

>
> IETF mail server glitch;  resending: Note: Robert already commented back
>
>

Robert,

Robert, Randy, Shane,

Senad

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 22 Jun 2012 16:34:31 +0000

Without being involved in the discussion whether it is the right problem to tackle or not, I would like to make the observation that at the very least, the draft needs to define a new BGP capability for the willingness to accept STALE routes. Peers that has not negotiated this capability MUST receive paths "as if" the STALE paths did not exist. This includes that STALE paths and any non-STALE paths whose nexthops resolve over a STALE paths MUST also not be sent to a such a peer. This way, negotiation of this capability by consenting adults will define an island where stale routes would roam free without affecting others.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Senad .Palislamovic
Sent: Friday, June 22, 2012 7:24 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Robert,

After going through all the emails, I admit, you've made a lot of valuable comments that 'seriously' MUST be considered in the new version of the draft.  However, putting that aside, I would really expect this to be saved for WG discussions.  I would think that at this stage, we are more less agreeing to take a look at this problem and its solution space.

Robert, Randy, Shane,

The main question I have for you is why object something that might not even affect you.  Draft's primary purpose is for the protection of the control plane for non-internet services (vpls, l2vpn, or infrustructure routes, i.e. BGP-3107), which in all cases does not affect you or me in any shape or form.  For folks to chose to extend this to l3vpn or even to IPv4/IPv6 space, they are doing it consciously knowing the nature of their network and its dynamics.  If you do see a route with the STALE community, and do not desire to use this path due its "unreliable" nature, you can request your provider to withdraw the routes with STALE community effectively making them DO_NOT_PERSIST at the source which effectively mimics the standard GR behavior; as Bruno stated, at eBGP, people are talking and negotiating.

So given all that, help me understand how could this impact anyone who chooses not to use it.  Am I missing anything?

Senad

On Wed, Jun 20, 2012 at 4:31 AM, Robert Raszuk <[email-masked]<mailto:[email-masked]>> wrote:

True. And changing it's fundamental behaviour every few months does not help to make it reliable/ performant either ;)

I am not asking for that at all. I am asking to simply use multiple implementations in the backend. Not two .. but 4 or 5. Probability that all fail/melt with the same bug I think is very very low.

Of course I admit that if you one uses service which only single vendor supports in BGP then one get's a bit stuck with the risk. And that seems to be one of the silent "problem statement" here.

Rgs,

R.

---

From: Enke Chen <[email-masked]>
Date: Fri, 22 Jun 2012 10:18:43 -0700

Hi, Saikat:

A new capability would certainly help.  It seems necessary, but may not 
be sufficient to limit the scope of the STALE path to "consenting 
adults".   For example, inside a network (i.e., AS), if there are 
routers that do not recognize the STALE community, a STALE path may 
still be advertised to external peers without the capability.   This 
means that all the routers in a network must recognize the community 
before enabling the capability on any of them.  It will be difficult to 
satisfy the condition universally.

In addition, to limit its scope the STALE community, once set, MUST not 
be removed by any receiver.

-- Enke

On 6/22/12 9:34 AM, Saikat Ray (sairay) wrote:
>
> Without being involved in the discussion whether it is the right 
> problem to tackle or not, I would like to make the observation that at 
> the very least, the draft needs to define a new BGP capability for the 
> willingness to accept STALE routes. Peers that has not negotiated this 
> capability MUST receive paths "as if" the STALE paths did not exist. 
> This includes that STALE paths and any non-STALE paths whose nexthops 
> resolve over a STALE paths MUST also not be sent to a such a peer. 
> This way, negotiation of this capability by consenting adults will 
> define an island where stale routes would roam free without affecting 
> others.
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *Senad .Palislamovic
> *Sent:* Friday, June 22, 2012 7:24 AM
> *To:* [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG 
> document - 3 more weeks
>
> valuable comments that 'seriously' MUST be considered in the new 
> version of the draft.  However, putting that aside, I would really 
> expect this to be saved for WG discussions.  I would think that at 
> this stage, we are more less agreeing to take a look at this problem 
> and its solution space.
>
> Robert, Randy, Shane,
>
> The main question I have for you is why object something that might 
> not even affect you.  Draft's primary purpose is for the protection of 
> the control plane for non-internet services (vpls, l2vpn, or 
> infrustructure routes, i.e. BGP-3107), which in all cases does not 
> affect you or me in any shape or form.  For folks to chose to extend 
> this to l3vpn or even to IPv4/IPv6 space, they are doing it 
> consciously knowing the nature of their network and its dynamics.  If 
> you do see a route with the STALE community, and do not desire to use 
> this path due its "unreliable" nature, you can request your provider 
> to withdraw the routes with STALE community effectively making them 
> DO_NOT_PERSIST at the source which effectively mimics the standard GR 
> behavior; as Bruno stated, at eBGP, people are talking and negotiating.
>
>     True. And changing it's fundamental behaviour every few months
>     does not help to make it reliable/ performant either ;)
>
>     I am not asking for that at all. I am asking to simply use
>     multiple implementations in the backend. Not two .. but 4 or 5.
>     Probability that all fail/melt with the same bug I think is very
>     very low.
>
>     vendor supports in BGP then one get's a bit stuck with the risk.
>     And that seems to be one of the silent "problem statement" here.
>
>     Rgs,
>
>
>     R.
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 17:27:28 +0000

Enke,

                The idea behind the setting of STALE is to inform speakers in other AS domains that the path is suspect as the session it was learned over is down.. IMO this is required to inform others about the viability of a path..

The other approach is to simply not mark these paths as STALE.. Simply de-pref in the local AS and business as usual when advertised outside the AS where all transitive attrs are reset..

This is exactly the semantic you are using in the GR draft. You do not de-pref the state locally and do not inform other AS domains that the paths learned over the session that GR is active on are suspect.

If the STALE CV is not honored across the AS border than the behavior defaults to what is specified in the GR draft. This should meet your requirements.

Jim Uttaro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Enke Chen
Sent: Friday, June 22, 2012 1:19 PM
To: Saikat Ray (sairay)
Cc: [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Hi, Saikat:

A new capability would certainly help.  It seems necessary, but may not be sufficient to limit the scope of the STALE path to "consenting adults".   For example, inside a network (i.e., AS), if there are routers that do not recognize the STALE community, a STALE path may still be advertised to external peers without the capability.   This means that all the routers in a network must recognize the community before enabling the capability on any of them.  It will be difficult to satisfy the condition universally.

In addition, to limit its scope the STALE community, once set, MUST not be removed by any receiver.

-- Enke

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Senad .Palislamovic
Sent: Friday, June 22, 2012 7:24 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 22 Jun 2012 17:30:54 +0000

Hi, Saikat:

A new capability would certainly help.  It seems necessary, but may not be sufficient to limit the scope of the STALE path to "consenting adults".   For example, inside a network (i.e., AS), if there are routers that do not recognize the STALE community,
[SR] That router must not negotiate the STALE capability with any peer, iBGP or eBGP, and hence should not receive any stale path from any router. The stale capability must be negotiated even within iBGP peers. To avoid inconsistency of iBGP routes in an AS, you might need to add the restriction that unless all iBGP peers negotiate the new capability, STALE paths should not be sent to any of them.

a STALE path may still be advertised to external peers without the capability.   This means that all the routers in a network must recognize the community before enabling the capability on any of them.  It will be difficult to satisfy the condition universally.

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 22 Jun 2012 13:36:23 -0400

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Saikat Ray (sairay)
Sent: Friday, June 22, 2012 10:31 AM
To: Enke Chen (enkechen)
Cc: [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Hi, Saikat:

A new capability would certainly help.  It seems necessary, but may not be sufficient to limit the scope of the STALE path to "consenting adults".   For example, inside a network (i.e., AS), if there are routers that do not recognize the STALE community,
[SR] That router must not negotiate the STALE capability with any peer, iBGP or eBGP, and hence should not receive any stale path from any router. The stale capability must be negotiated even within iBGP peers. To avoid inconsistency of iBGP routes in an AS, you might need to add the restriction that unless all iBGP peers negotiate the new capability, STALE paths should not be sent to any of them.
[Jakob Heitz] What if a Route Reflector acquires a client without the capability?

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 22 Jun 2012 13:46:03 -0400

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of UTTARO, JAMES
Sent: Friday, June 22, 2012 10:27 AM
To: 'Enke Chen'; Saikat Ray (sairay)
Cc: [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Enke,

                The idea behind the setting of STALE is to inform speakers in other AS domains that the path is suspect as the session it was learned over is down.. IMO this is required to inform others about the viability of a path..
[Jakob Heitz]  Then call them SUSPECT instead of STALE.

The other approach is to simply not mark these paths as STALE.. Simply de-pref in the local AS and business as usual when advertised outside the AS where all transitive attrs are reset..
[Jakob Heitz]  SUSPECT is entirely different to low preference.
A low preference path exists. A SUSPECT path may not exist.

A less specific path (shorter netmask) exists and will forward packets,
unless it is the default route or also SUSPECT.
This puts a SUSPECT route at an equal or lower preference than a default route.

Therefore a SUSPECT path should be de-preferred among the
set of ALL available paths, including the less specific ones.

---

From: Enke Chen <[email-masked]>
Date: Fri, 22 Jun 2012 10:48:18 -0700

Jim,

With GR, as you correctly pointed out, a path would not stale for many 
hours or days. That is why it may not be as critical to limit the scope 
of the stale path.

If we were to have long-lived stale paths (which I think is a bad idea), 
I agree that they should be limited to "consenting adults", and there 
must be some mechanisms (such as capability) in place.

-- Enke

On 6/22/12 10:27 AM, UTTARO, JAMES wrote:
>
> Enke,
>
>                 The idea behind the setting of STALE is to inform 
> speakers in other AS domains that the path is suspect as the session 
> it was learned over is down.. IMO this is required to inform others 
> about the viability of a path..
>
> The other approach is to simply not mark these paths as STALE.. Simply 
> de-pref in the local AS and business as usual when advertised outside 
> the AS where all transitive attrs are reset..
>
> This is exactly the semantic you are using in the GR draft. You do not 
> de-pref the state locally and do not inform other AS domains that the 
> paths learned over the session that GR is active on are suspect.
>
> If the STALE CV is not honored across the AS border than the behavior 
> defaults to what is specified in the GR draft. This should meet your 
> requirements.
>
> Jim Uttaro
>
> Of *Enke Chen
> *Sent:* Friday, June 22, 2012 1:19 PM
> *To:* Saikat Ray (sairay)
> *Cc:* [email-masked]
> A new capability would certainly help.  It seems necessary, but may 
> not be sufficient to limit the scope of the STALE path to "consenting 
> before enabling the capability on any of them.  It will be difficult 
> to satisfy the condition universally.
>
> *From:[email-masked] <mailto:[email-masked]> 
> [mailto:[email-masked]] *On Behalf Of *Senad .Palislamovic
> *To:* [email-masked] <mailto:[email-masked]>
> *Cc:* [email-masked] <mailto:[email-masked]>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 18:03:33 +0000

Enke,

                I think we have got to have a consistent approach to these solutions.. I am not comfortable with the fact that it is "ok" for GR but not "ok" for Persistence based on a IMO subjective interpretation of the amount of time a STALE path is active in local and adjacent topologies. An hour is a long time, and as GR is used primarily used n IPV4 which is dynamic it would seem that a) these paths should be de-pref after some amount of time b) GR should inform other AS topologies of the staleness by using the STALE CV..

I would be interested in hearing from operators as to if this behavior is understood and if limiting of the time to ~1hr makes the inconsistency within and across AS domains acceptable..  In your opinion what is the maximum acceptable amount of time that we could use Persistence.

Jim Uttaro

From: Enke Chen [mailto:[email-masked]]
Sent: Friday, June 22, 2012 1:48 PM
To: UTTARO, JAMES
Cc: Saikat Ray (sairay); [email-masked]; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

With GR, as you correctly pointed out, a path would not stale for many hours or days. That is why it may not be as critical to limit the scope of the stale path.

If we were to have long-lived stale paths (which I think is a bad idea), I agree that they should be limited to "consenting adults", and there must be some mechanisms (such as capability) in place.

-- Enke

Jim Uttaro

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Enke Chen
Sent: Friday, June 22, 2012 1:19 PM
To: Saikat Ray (sairay)
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Hi, Saikat:

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: "Saikat Ray (sairay)" <[email-masked]>
Date: Fri, 22 Jun 2012 22:38:14 +0000

That client will not get any STALE route ;)

---

From: Shane Amante <[email-masked]>
Date: Fri, 22 Jun 2012 19:59:39 -0600

On Jun 22, 2012, at 12:03 PM, UTTARO, JAMES wrote:

> Enke,
>  
>                 I think we have got to have a consistent approach to these solutions.. I am not comfortable with the fact that it is “ok” for GR but not “ok” for Persistence based on a IMO subjective interpretation of the amount of time a STALE path is active in local and adjacent topologies. An hour is a long time, and as GR is used primarily used n IPV4 which is dynamic it would seem that a) these paths should be de-pref after some amount of time b) GR should inform other AS topologies of the staleness by using the STALE CV..
>  

As an operator I believe even 1 hour is too long.  In fact, I'd rather (and, I do) trust Non-Stop Routing (NSR) on PE's in the network, because I know that CE's will not understand GR capability and will not be able to benefit from continuing to forward packets even while the PE is restarting.  Granted, this is an environment where there are many "unmanaged CE's", where I have no control/say over what SW or the capabilities are on those CE's.

Furthermore, with control plane redundancy for iBGP sessions from one PE across multiple RR's serving that cluster, it is possible to conduct maintenance on one half of the RR-pair serving a cluster while still preserving control and/or forwarding plane behavior.

Ultimately, as I've said before, I'd much rather not see any type of "stale" routes 'sticking around' in the control plane, since that is a vast change in behavior from today's deployment of BGP where reachability information is only carried while there are active BGP sessions.  (Yes, I get your point that I don't have to use persistence if I don't want to, but if developers are in there monkey'ing around with code to add this feature, inevitably I pay a price in addt'l complexity for this "feature" even if I don't turn it on, i.e.: there are potentially 'latent' bugs caused by this "feature", which is not acceptable).

As Enke has stated previously, I believe this class of problem is a rare one, (given a proper network architecture), and thus we should not be creating AFI-specific solutions that add substantial complexity to an already complex protocol. 

-shane

> Without being involved in the discussion whether it is the right problem to tackle or not, I would like to make the observation that at the very least, the draft needs to define a new BGP capability for the willingness to accept STALE routes. Peers that has not negotiated this capability MUST receive paths “as if” the STALE paths did not exist. This includes that STALE paths and any non-STALE paths whose nexthops resolve over a STALE paths MUST also not be sent to a such a peer. This way, negotiation of this capability by consenting adults will define an island where stale routes would roam free without affecting others.
>

---

From: Randy Bush <[email-masked]>
Date: Fri, 22 Jun 2012 18:00:57 -1000

> As Enke has stated previously, I believe this class of problem is a
> rare one, (given a proper network architecture), and thus we should
> not be creating AFI-specific solutions that add substantial complexity
> to an already complex protocol.

what he said

randy

---

From: "Neil J. McRae" <[email-masked]>
Date: Sat, 23 Jun 2012 09:40:57 +0000

I support this work also. We can no longer have the entire control plane just give up. Some folks have labeled this as rare. In comparison with other IDR type problems, I have seen this issue more than any other problem and given the catastrophic effect (I.e. complete and total network outage) and the impact that has not just in Internet space but also VPN space where the impact of a complete network loss on critical infrastructure is not acceptable (hospitals, law enforcement and other important services). In every case where this has happened there has never been an actual reachability problem either.

From: <Henderickx>, "Wim (Wim)" <[email-masked]<mailto:[email-masked]>>
Date: Monday, 18 June 2012 19:15
To: Paul Unbehagen <[email-masked]<mailto:[email-masked]>>, "Fragassi, Roberto (Roberto)" <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, Susan Hares <[email-masked]<mailto:[email-masked]>>, "'John G. Scudder'" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

+1

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Paul Unbehagen
Sent: maandag 18 juni 2012 16:39
To: Fragassi, Roberto (Roberto)
Cc: [email-masked]<mailto:[email-masked]>; 'John G. Scudder'; Susan Hares
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

+1

--
Paul

+1 Support.

IDR WG and BGPers:

Sue Hares

---

From: John Leslie <[email-masked]>
Date: Sat, 23 Jun 2012 07:58:17 -0400

Neil J. McRae <[email-masked]> wrote:
> 
> I support this work also. We can no longer have the entire control plane
> just give up. Some folks have labeled this as rare. In comparison with
> other IDR type problems, I have seen this issue more than any other
> problem and given the catastrophic effect (I.e. complete and total
> network outage) and the impact that has not just in Internet space but
> also VPN space where the impact of a complete network loss on critical
> infrastructure is not acceptable (hospitals, law enforcement and other
> important services). In every case where this has happened there has
> never been an actual reachability problem either.

   Thank you for this level of detail as your reason to support this as
a WG draft. It is much more helpful than the too-brief "I support" posts.

   Some of us, however, seeing the same problem look to operational fixes,
not protocol changes. I, for example, like to maintain redundant BGP
sessions over the same link with one kept particularly simple. This can
be done in a single day, instead of waiting many months for a standards
change and more months for equipment to reach the field. It also keeps
everything under my control, rather than passing the buck to my
competitors to ensure that stale routes actually get withdrawn.

--
John Leslie <[email-masked]>

---

From: "Neil J. McRae" <[email-masked]>
Date: Sat, 23 Jun 2012 12:44:47 +0000

John,
Thanks for your feedback but in reality this work around won't help
especially in large VPN networks but even in just plain old internet land
With the probability of this issue occurring is increasing and not
decreasing.

And btw its not months the industry has been waiting for this to be
resolved. Its more than fifteen years. We need to act to make the control
plane more resilient, I'm not sure if this is the best solution but its
something that is worthy exploration given the catastrophic result of
failure. What I find somewhat incredible is that any vendor that had
survivability of this nature would fly to the top of our selection process!

When we do so much in other protocols to ensure reliability it seems
incredible that we choose not to here, and as I said, in all cases of this
scenario impacting me, and what leads me to believe is crucially a flaw in
the current protocol, reachability has always been present, which is what
our protocol is all about.

Regards,
Neil.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sat, 23 Jun 2012 19:28:13 +0000

+1.. IMO I think that folks need to come to the realization that BGP is being used in varied use cases, using the internet application as the only frame of reference or lens into the requirements for reliability is simply not appropriate or correct.. I can tell you as someone who actually runs networks that this is not rare and has been seen in multiple services.. It needs to be fixed. As Neil points out the VPN/VPLS cases are used to support many services that cannot afford the luxury of being down..

Jim Uttaro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Neil J. McRae
Sent: Saturday, June 23, 2012 5:41 AM
To: Henderickx, Wim (Wim); Paul Unbehagen; Fragassi, Roberto (Roberto)
Cc: [email-masked]; 'John G. Scudder'; Susan Hares
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

+1

+1

--
Paul

+1 Support.

IDR WG and BGPers:

Sue Hares

---

From: <[email-masked]>
Date: Sat, 23 Jun 2012 19:43:13 +0000

Shane, all,

From: Shane Amante Sent: Saturday, June 23, 2012 4:00 AM

Enke,

Additional complexity is a valid argument, but you would need to evaluate it for all features. e.g. As per your own argument, one could say that Non-Stop Routing "is not acceptable".

Also,  it might be useful to remember that the question is about WG adoption, not implementation. In the context of this draft, this is largely independent:
- a WG doc may never be implemented
- to some extent BGP persistence could be implemented without WG adoption.

If one SP asked BGP persistence implementation to a vendor, then only both of them would define BGP persistence. IDR would not have the opportunity to comment/improve/delay/kill (pick your own) the proposal.
Given the importance of the subject, I would prefer that IDR had the ability to have is word on this.

Bruno

-shane

Jim Uttaro

From: Enke Chen [mailto:[email-masked]]
Sent: Friday, June 22, 2012 1:48 PM
To: UTTARO, JAMES
Cc: Saikat Ray (sairay); [email-masked]<mailto:[email-masked]>; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

-- Enke

Jim Uttaro

Hi, Saikat:

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sat, 23 Jun 2012 19:45:39 +0000

Shane,

                First of all if it was as easy as redundant BGP sessions or as you state later "proper network architecture" we would not be having this conversation.. Obvious network architecture like redundant sessions so I can perform maintenance mode is pretty much the starting point of a robust architecture across AFs/Services and I would hazard a guess that everyone does this.. Beyond that there are methods to enhance reliability even further that I have taken ( We could have a pvt conversation if you would like ).  All of this being said, there are implementation bugs, overloading of BGP machinery etc.. that have occurred and will occur in the future.. We cannot simply shrug and say it is not important as it in some folks mind rare ( not accurate ) or that there is some magical network architecture that if only everyone would use, this issue could be mitigated. IMO it is the job of the IETF to address real problems in the real world..

Back to this rare nonsense.. Rob Shakir wrote a draft to correct what I guess is a rare condition having to do with a mal-formed attr. He is very descriptive in terms f syntactic and semantic bugs. Folks in IDR including E.Chen is an author of a solutions draft. If this is so rare why is there a req and solutions draft..

We cannot have the argument coming out of both sides of the mouth..

Jim Uttaro

From: Shane Amante [mailto:[email-masked]]
Sent: Friday, June 22, 2012 10:00 PM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Enke,

-shane

Jim Uttaro

Jim,

-- Enke

Jim Uttaro

Hi, Saikat:

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: <[email-masked]>
Date: Sat, 23 Jun 2012 19:52:07 +0000

Hi Enke,

From: Enke Chen Sent: Friday, June 22, 2012 7:19 PM

Hi, Saikat:

I'm not sure to see the real difficulty. This does not require a new implementation but only to update a routemap. ISP already widely use communities.

Alternatively, the router activating persistence and setting the STALE community could also set the NO_EXPORT community. Hence by default the STALE route would not be advertised to other ASes.

Bruno

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 23 Jun 2012 22:00:08 +0200

> Alternatively, the router activating persistence and setting the STALE
> community could also set the NO_EXPORT community. Hence by default the
> STALE route would not be advertised to other ASes.

That would be terribly broken I am afraid. Dropping the route at PE/ASBR 
(what NO_EXPORT does) does not mean generating a withdraw.

So all of your EBGP peers would never get the withdraw during the 
persistance period which as the draft say can be infinity. The end 
effect would be completely broken routing.

r.

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 23 Jun 2012 22:14:43 +0200

Jim,

> IMO I think that folks need to come to the realization that BGP is
> being used in varied use cases, using the internet application as the
> only frame of reference or lens into the requirements for reliability is
> simply not appropriate or correct..

+

> All of this being said, there are implementation bugs, overloading of
 > BGP machinery etc.. that have occurred and will occur in the future..

I think you need to realize that the key for any application is solid IP 
reachability.

You can realize any VPN to your customers without BGP being used as a 
service bus for it. It is your choice or your vendor's choice as you say 
to "overload BGP machinery" for everything.

Putting more and more questionable requirements on the only protocol 
which provides such global IP reachability proves as a bad idea. It 
makes the protocol more and more fragile.

The motivation that various flavors of VPNs carried by BGP must be up 
only proves that as number of people suggested in the past the same BGP 
protocol should not be used for everything.

The point is not that protocol will not be able to cope with the 
requirement. The point is that it is wrong to keep changing it's core 
behaviour because your VPN or VPLS service goes down.

Moreover you now do not want to limit the changes only to your services 
SAFI, but argue to "enhance" all AFI/SAFIs with it. That is something 
very hard to agree with.

R.

---

From: John Leslie <[email-masked]>
Date: Sat, 23 Jun 2012 18:12:02 -0400

> especially in large VPN networks but even in just plain old internet land

   Obviously, my network is not the same as yours -- I didn't mean to
imply that my solution would work for your network, merely that if I
were facing your network problems I would be looking for an operational
solution. (I'd be _quite_ surprised if there isn't one to make the routes
propagated exactly what you want, but under _your_ control.)

> With the probability of this issue occurring is increasing and not

   That is, alas, true.

> plane more resilient...

   To do that, we must be willing to say "No" to folks who propose to
introduce new bugs. Alas, the -01 version of bgp-persistence introduces
entirely too many bugs for my taste. I am willing to change my opinion
if an -02 version gets rid of them, but I currently don't believe this
is likely.

> incredible that we choose not to here,

   Reliability is first a matter of leaving out the bugs we can leave
out, and second a matter of gathering the operational evidence for bugs
that show up.

   When operators aren't willing to read the logs (but prefer to wait
for sessions to go down) I really don't see how to make much progress.

> and as I said, in all cases of this scenario impacting me, and what
> leads me to believe is crucially a flaw in the current protocol,
> reachability has always been present, which is what our protocol is
> all about.

   Alas, that's not quite true. BGP is about exchanging routing
information.  If it's working right, that gives us good information
about reachability. Good, not perfect, alas...

   Reachability, in a theoretical sense, can best be deterimined by
a flooding algorithm to see what works. (There are bunches of reasons
why we don't do it that way.)

   There's nothing wrong with you remembering a stale route your peer
advertised and trying it. What's wrong is to lead your other peers to
believe it's still being advertised to you. BGP _could_ have been
designed to make it reasonable for you to advertise routes which are
neither your own nor are advertised to you; but it wasn't. When this
happens in today's Internet, we call it a bug and prevail upon you to
stop.

   I freely admit I don't understand exactly what problem you're trying
to solve -- except that it involves you wanting some of your peers to
use stale routes "because they work". Alas, that's not the BGP paradigm:
BGP wants routes that are advertised over the same links that are being
used to reach the destination, and BGP _doesn't_ want routers to have
any responsibility to determine if routes "work" (whatever that means).

   I don't know whether the IDR WG _can_ offer you a solution. Old-timers
fundamentally don't _want_ to deterimine actual reachability via routers
with fried control planes but working data planes. Perhaps if we
understood your needs better we could advise on how to advertise
"reachability" in the absence of routing information; but right now we
suspect you'll cause more problems than you solve.

---

From: Enke Chen <[email-masked]>
Date: Sat, 23 Jun 2012 15:33:03 -0700

Jim,

Let us remain civil in the discussion. If you want to use the word 
"nonsense", I suggest that you go back to read some of you posts.

I have never said that "malform attributes" are rare.  Just the 
contrary, the issue of session reset from malformed attributes has been 
with us for many years.  There have been large outages from time to 
time.  There is a major bug in the base bgp spec in that area, in 
particular, regarding the handling of optional transitive attribute.  
That is what the error handling draft is trying to address.

-- Enke

On 6/23/12 12:45 PM, UTTARO, JAMES wrote:
>
> Shane,
>
>                 First of all if it was as easy as redundant BGP 
> sessions or as you state later "proper network architecture" we would 
> not be having this conversation.. Obvious network architecture like 
> redundant sessions so I can perform maintenance mode is pretty much 
> the starting point of a robust architecture across AFs/Services and I 
> would hazard a guess that everyone does this.. Beyond that there are 
> methods to enhance reliability even further that I have taken ( We 
> could have a pvt conversation if you would like ).  All of this being 
> said, there are implementation bugs, overloading of BGP machinery 
> etc.. that have occurred and will occur in the future.. We cannot 
> simply shrug and say it is not important as it in some folks mind rare 
> ( not accurate ) or that there is some magical network architecture 
> that if only everyone would use, this issue could be mitigated. IMO it 
> is the job of the IETF to address real problems in the real world..
>
> Back to this rare nonsense.. Rob Shakir wrote a draft to correct what 
> I guess is a rare condition having to do with a mal-formed attr. He is 
> very descriptive in terms f syntactic and semantic bugs. Folks in IDR 
> including E.Chen is an author of a solutions draft. If this is so rare 
> why is there a req and solutions draft..
>
> *From:*Shane Amante [mailto:[email-masked]]
> *Sent:* Friday, June 22, 2012 10:00 PM
> *To:* UTTARO, JAMES
> *Cc:* 'Enke Chen'; [email-masked]
>                 I think we have got to have a consistent approach to 
> these solutions.. I am not comfortable with the fact that it is "ok" 
> for GR but not "ok" for Persistence based on a IMO subjective 
> interpretation of the amount of time a STALE path is active in local 
> and adjacent topologies. An hour is a long time, and as GR is used 
> primarily used n IPV4 which is dynamic it would seem that a) these 
> paths should be de-pref after some amount of time b) GR should inform 
> other AS topologies of the staleness by using the STALE CV..
>
> I would be interested in hearing from operators as to if this behavior 
> is understood and if limiting of the time to ~1hr makes the 
> inconsistency within and across AS domains acceptable..  In your 
> opinion what is the maximum acceptable amount of time that we could 
> use Persistence.
>
> As an operator I believe even 1 hour is too long.  In fact, I'd rather 
> (and, I do) trust Non-Stop Routing (NSR) on PE's in the network, 
> because I know that CE's will not understand GR capability and will 
> not be able to benefit from continuing to forward packets even while 
> the PE is restarting.  Granted, this is an environment where there are 
> many "unmanaged CE's", where I have no control/say over what SW or the 
> capabilities are on those CE's.
>
> Furthermore, with control plane redundancy for iBGP sessions from one 
> PE across multiple RR's serving that cluster, it is possible to 
> conduct maintenance on one half of the RR-pair serving a cluster while 
> still preserving control and/or forwarding plane behavior.
>
> Ultimately, as I've said before, I'd much rather not see any type of 
> "stale" routes 'sticking around' in the control plane, since that is a 
> vast change in behavior from today's deployment of BGP where 
> reachability information is only carried while there are active BGP 
> sessions.  (Yes, I get your point that I don't have to use persistence 
> if I don't want to, but if developers are in there monkey'ing around 
> with code to add this feature, inevitably I pay a price in addt'l 
> complexity for this "feature" even if I don't turn it on, i.e.: there 
> are potentially 'latent' bugs caused by this "feature", which is not 
> acceptable).
>
> *From:*Enke Chen [mailto:[email-masked]]
> *Sent:*Friday, June 22, 2012 1:48 PM
> *To:*UTTARO, JAMES
> *Cc:*Saikat Ray (sairay);[email-masked] <mailto:[email-masked]>; Enke Chen
> *Subject:*Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG 
> document - 3 more weeks
>
> Jim,
>
> hours or days. That is why it may not be as critical to limit the 
> scope of the stale path.
>
> If we were to have long-lived stale paths (which I think is a bad 
> idea), I agree that they should be limited to "consenting adults", and 
> there must be some mechanisms (such as capability) in place.
>
> -- Enke
>
> *From:[email-masked] 
> <mailto:[email-masked]>[mailto:[email-masked]]*On Behalf 
> Of*Enke Chen
> *Sent:*Friday, June 22, 2012 1:19 PM
> *To:*Saikat Ray (sairay)
> *Cc:[email-masked] <mailto:[email-masked]>
> *Subject:*Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG 
> document - 3 more weeks
>
> Hi, Saikat:
>
> *From:[email-masked] 
> <mailto:[email-masked]>[mailto:[email-masked]]*On Behalf 
> Of*Senad .Palislamovic
> *Sent:*Friday, June 22, 2012 7:24 AM
> *To:[email-masked] <mailto:[email-masked]>
> *Cc:[email-masked] <mailto:[email-masked]>
> *Subject:*Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG 
> document - 3 more weeks
>

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 24 Jun 2012 00:32:36 +0200

> (what NO_EXPORT does) does not mean generating a withdraw.

I'll take it back.

In order to verify I have tested this in my lab and no matter what I try 
the routers/ios version I am using do not let me set NO_EXPORT (or any 
other community) on the IBGP in or out sessions (PE or RR).

If I set it on inbound EBGP sessions (route-map in) it works as it 
supposed to and withdraw is generated.

So after all I am not sure if this is applicable to be used by the 
persistence as a workaround to limit the advertisement radius of STALE 
paths.

R.

---

From: Enke Chen <[email-masked]>
Date: Sat, 23 Jun 2012 15:47:06 -0700

Hi, Neil:

Thanks for sharing your thoughts and experiences.  That is certainly 
helpful to the discussion.

Over the years I have personally experienced and witnessed a number of 
major network outages that were bgp related.  Most of them, if not all, 
were caused by malformed updates.  That's why the work on error handling 
started in both IDR and GROW WGs.

It will be really valuable if you could share with us whether the 
revised error handling would address issues that you encountered in your 
network:

http://datatracker.ietf.org/doc/draft-ietf-idr-error-handling/?include_text=1

Regards,  -- Enke

On 6/23/12 5:44 AM, Neil J. McRae wrote:
> On 23/06/2012 12:58, "John Leslie"<[email-masked]>  wrote:
>> John Leslie<[email-masked]>
>>
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sun, 24 Jun 2012 02:29:59 +0000

Enke,

                I agree lets be civil..

So civility starts with the following:

I say I have a real problem and you do not dismiss it as a rare occurrence and therefore not warranting WG adoption for the problem to be worked on.. Do you think I am lying or lack the ability to count the number of times it has happened and how many major/catastrophic outages it has caused.. It is not appropriate to discuss the outages in this venue or their cause..

I say I have a real problem and it is not dismissed as Jim Uttaro does not know how to properly architect his network for robustness, reliability, and survivability. Gee If he could just figure that out ( It is obvious ) there would be no problem.

You address the very real fact that a solution GR has been adopted and coded that has exactly the behavior your are stating is not acceptable for the Persistence draft which BTW has the STALE CV to address this exact deficiency of not informing topologies of STALE paths which is the current behavior of GR. Stating that it is an hour, therefore OK does not cut it..

Folks acknowledge that BGP is being used for other services and they merit the same attention as the traditional internet application.. To this point it seems that IDR may be the wrong place for this draft and it may be more appropriate in L3VPN, L2VPN. Maybe we should consider that other AFs i.e RT-C, or other apps flowspec etc...should be moved into L3VPN as it is probably a more appropriate place..

I do want to address the Update Error condition(s).. As I responded to Rob, I think we need an accounting of how this behavior when coded would or would not be applicable to other AFs. I would prefer a consistent BGP code base across all AFs.  We have to be careful that we do not have so many responses to error conditions. It is difficult enough to train Ops/Support staff, develop tools across AFs/Service and get features consistently developed..

Jim Uttaro

From: Enke Chen [mailto:[email-masked]]
Sent: Saturday, June 23, 2012 6:33 PM
To: UTTARO, JAMES
Cc: 'Shane Amante'; [email-masked]; Enke Chen
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

Let us remain civil in the discussion. If you want to use the word "nonsense", I suggest that you go back to read some of you posts.

I have never said that "malform attributes" are rare.  Just the contrary, the issue of session reset from malformed attributes has been with us for many years.  There have been large outages from time to time.  There is a major bug in the base bgp spec in that area, in particular, regarding the handling of optional transitive attribute.  That is what the error handling draft is trying to address.

-- Enke

Jim Uttaro

From: Shane Amante [mailto:[email-masked]]
Sent: Friday, June 22, 2012 10:00 PM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Enke,

-shane

Jim Uttaro

Jim,

-- Enke

Jim Uttaro

Hi, Saikat:

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: Enke Chen <[email-masked]>
Date: Sat, 23 Jun 2012 19:58:49 -0700

Jim:

On 6/23/12 7:29 PM, UTTARO, JAMES wrote:
> I say I have a real problem and you do not dismiss it as a rare 
> occurrence and therefore not warranting WG adoption for the problem to 
> be worked on.. Do you think I am lying or lack the ability to count 
> the number of times it has happened and how many major/catastrophic 
> outages it has caused.. It is not appropriate to discuss the outages 
> in this venue or their cause..
>

It is not appropriate for me to talk about another person's character or 
ability at IDR.  I speak based on my experience and judgement, not on 
another person's character/ability.

> I say I have a real problem and it is not dismissed as Jim Uttaro does 
> not know how to properly architect his network for robustness, 
> reliability, and survivability. Gee If he could just figure that out ( 
> It is obvious ) there would be no problem.
>
> You address the very real fact that a solution GR has been adopted and 
> coded that has exactly the behavior your are stating is not acceptable 
> for the Persistence draft which BTW has the STALE CV to address this 
> exact deficiency of not informing topologies of STALE paths which is 
> the current behavior of GR. Stating that it is an hour, therefore OK 
> does not cut it..
>

This has been hashed multiple times, hasn't it?

Again, GR allows stale paths to be kept for a short duration.  I do not 
feel comfortable to have stale paths in the routing system for hours or 
days.  That would make BGP semi-static and semi-dynamic.  The 
implication is non-trivial, as pointed by others.

> Folks acknowledge that BGP is being used for other services and they 
> merit the same attention as the traditional internet application.. To 
> this point it seems that IDR may be the wrong place for this draft and 
> it may be more appropriate in L3VPN, L2VPN. Maybe we should consider 
> that other AFs i.e RT-C, or other apps flowspec etc...should be moved 
> into L3VPN as it is probably a more appropriate place..
>
> I do want to address the Update Error condition(s).. As I responded to 
> Rob, I think we need an accounting of how this behavior when coded 
> would or would not be applicable to other AFs. I would prefer a 
> consistent BGP code base across all AFs.  We have to be careful that 
> we do not have so many responses to error conditions. It is difficult 
> enough to train Ops/Support staff, develop tools across AFs/Service 
> and get features consistently developed..
>

Based on my experience and judgement, I believe that the error handling 
for malformed updates is generally applicable to all the address 
families.  I think that several folks including JGS have explained very 
well why GR or persistence would not be the right solution in these cases.

-- Enke

> *Sent:* Saturday, June 23, 2012 6:33 PM
> *To:* UTTARO, JAMES
> *Cc:* 'Shane Amante'; [email-masked]; Enke Chen
> contrary, the issue of session reset from malformed attributes has 
> been with us for many years.  There have been large outages from time 
> to time.  There is a major bug in the base bgp spec in that area, in 
> *Cc:* 'Enke Chen'; [email-masked] <mailto:[email-masked]>

---

From: Randy Bush <[email-masked]>
Date: Sat, 23 Jun 2012 18:49:00 -1000

> in this venue or their cause..

then how do we make the engineering tradeoffs to understand if this
complex and somewhat dangerous hack will ameliorate them?

randy

---

From: heasley <[email-masked]>
Date: Sun, 24 Jun 2012 06:35:16 +0000

> is down.  Perhaps someone can enlighten me. 

i think collective-your time would be better spent developing an open source
test suite that all implementers can use test their crap, than this and/or
grow-ops-reqs-for-bgp-error-handling.

---

From: Rob Shakir <[email-masked]>
Date: Sun, 24 Jun 2012 10:15:24 +0100

> grow-ops-reqs-for-bgp-error-handling.

Whilst further testing is of course useful, and likely to find some further issues, I think that any implementor and/or network operator will attest to fact that testing does not find every issue. Essentially, even if it does, we just patch them on an instance-by-instance basis - whilst significant impact is incurred in live networks in the meantime. 

I'm absolutely for more testing, but it seems to me like where we can look at a solution that means that we impact less when we do get errors occurring (such as affecting only particular NLRI etc.), this is of value.

Kind regards,
r.

---

From: Shane Amante <[email-masked]>
Date: Sun, 24 Jun 2012 13:04:56 -0600

Jim,

Please note that you asked for operator input.  I am providing that and, in doing so, I am not intending to cast aspersions on you or anyone else.  I am only attempting to share my knowledge and, more importantly, my operational experience.  See below.

On Jun 23, 2012, at 1:45 PM, UTTARO, JAMES wrote:
> Shane,
>  
>                 First of all if it was as easy as redundant BGP sessions or as you state later “proper network architecture” we would not be having this conversation.. Obvious network architecture like redundant sessions so I can perform maintenance mode is pretty much the starting point of a robust architecture across AFs/Services and I would hazard a guess that everyone does this.. Beyond that there are methods to enhance reliability even further that I have taken ( We could have a pvt conversation if you would like ).  All of this being said, there are implementation bugs, overloading of BGP machinery etc.. that have occurred and will occur in the future.. We cannot simply shrug and say it is not important as it in some folks mind rare ( not accurate ) or that there is some magical network architecture that if only everyone would use, this issue could be mitigated. IMO it is the job of the IETF to address real problems in the real world..

Here's the problem I have with the above.  From the vantage point of my __VPN__ networks (IPVPN, VPLS), I am not observing the same types of network failures (in those networks) as you have observed.  Please note that I am not saying your claims are false, I am only stating that this these types of outages have not occurred in my VPN networks.  As such, I also think it's important for the WG to understand this additional data point from the vantage point of a wholly different set of VPN networks, so that WG members can see a "balanced view".  I don't know (and, likely will never know) why we are not seeing these problem(s) and you are, but I imagine that it's likely due to some combination of, and in no particular order: different HW, different SW and, potentially, different scale.  Ultimately, this operational experience is why I'm stating that this _class_ of problem, specifically related to VPN networks, is very rare.

> Back to this rare nonsense.. Rob Shakir wrote a draft to correct what I guess is a rare condition having to do with a mal-formed attr. He is very descriptive in terms f syntactic and semantic bugs.…

I have read and do support Rob's work.  However, to be perfectly clear, I do believe that we need to solve this overall set of problems (as defined in Rob's draft), just not as has been proposed in the BGP persistence draft.  More to the point, where I __have__ observed this class of problem is related to the Internet side, (AFI/SAFI 1/1 to be more precise), during a handful of occasions over the last several years.  However, most (but, certainly not all) of those failures were clearly attributable to implementation failures (on, the receive side of receiving an UPDATE), for which I don't believe that any solution the IDR WG devises will ever (or, should ever) be able to fix at the end of the day.  So, this ultimately boils down to (again, on the Internet-side) a small number of very large disruptions to the control plane as a result of malformed attributes.  Again, yes, we should solve those.  So, forgive me for being selfish here, but this is why I would rather see the IDR WG solve this in a generic way for all AFI's, so at least if we're going to add complexity to implementations we're _all_ gaining the benefit from it.

> We cannot have the argument coming out of both sides of the mouth..

I'm sorry if you feel that way, but again I am only stating that my experience has been different from yours.  Obviously, each WG member is free to draw their own conclusions as to those individual data points.

-shane

---

From: Hannes Gredler <[email-masked]>
Date: Mon, 25 Jun 2012 09:07:30 +0200

robert,

On Sat, Jun 23, 2012 at 10:14:43PM +0200, Robert Raszuk wrote:
| >IMO I think that folks need to come to the realization that BGP is
| >being used in varied use cases, using the internet application as the
| >only frame of reference or lens into the requirements for reliability is
| >simply not appropriate or correct..
| 
| +
| 
| >All of this being said, there are implementation bugs, overloading of
| > BGP machinery etc.. that have occurred and will occur in the future..
| 
| 
| I think you need to realize that the key for any application is
| solid IP reachability.

well, there are some of us who believe that there are other ways
of building a service architecture and (surprise !) the transport
path not necessarily has to be IP.

| You can realize any VPN to your customers without BGP being used as
| a service bus for it. It is your choice or your vendor's choice as
| you say to "overload BGP machinery" for everything.
             ^^^^^^^^^^^^^^^^^^^^^^^

is is really a smart thing to question over-and-over things that:
  a. have been decided 10+ years ago
  b. have proven to be useful
 
| Putting more and more questionable requirements on the only protocol
| which provides such global IP reachability proves as a bad idea. It
| makes the protocol more and more fragile.

i'd consider the questionw around
  a. graceful control-plane restart and
  b. standardized handling of forwarding entries thereafter

as something that should be answered for any protocol in the design phase;
obviously BGP is lacking this and that's why we're getting our
heads together ...

| The motivation that various flavors of VPNs carried by BGP must be
| up only proves that as number of people suggested in the past the
| same BGP protocol should not be used for everything.
|
| The point is not that protocol will not be able to cope with the
| requirement. The point is that it is wrong to keep changing it's
| core behaviour because your VPN or VPLS service goes down.
|
| Moreover you now do not want to limit the changes only to your
| services SAFI, but argue to "enhance" all AFI/SAFIs with it. That is
| something very hard to agree with.

the "core behaviour" of the protocol (kill RIB entries right
after session loss) is wrong. and since BGP has outgrown its
original purpose (route 1K prefixes) vs. route 100Ks of prefixes,
route unicast and multicast, route VPN/non-VPN,
route non-routing related functionality it is just letigimate to discuss
restart and persistence behaviour on a per AFI/SAFI basis.

tx,

/hannes

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 25 Jun 2012 10:26:04 +0200

Hi Hannes,

> path not necessarily has to be IP.

Well actually above I was not intending to sound anti-MPLS. I should say 
IP/MPLS above .. as you very well know MPLS requires IP to work first !

> | you say to "overload BGP machinery" for everything.

I think it is.

It is not about putting services in BGP or not as the protocol. I do not 
have any problem with that at all. Note that "overload" term was a quote 
from Jim's mail no my own description.

It is about running services BGP and IP BGP together which I am 
questioning. Keyur and I had a draft http://goo.gl/GWrIs "Transport 
Instance BGP" to allow for clear separation of the two instances.

I do believe that if we would have this in place there would be 
completely no objection from anyone in IETF to add persistence to 
"Service BGP". Especially that Bruno mentioned the need to propagate his 
infrastructure routes in SAFI 1/1. Both BGP instances could operate 
independently and "IP/Routing instance" could serve very well the 
"service instance".

Yes I realize that for you it means more code to maintain so there is 
price to that. The only question I think which should be stated if 
perhaps cost vs benefit of such separation is getting to right more now.

> obviously BGP is lacking this and that's why we're getting our

Hmmmm one would think that RFC4724 addresses that. And moreover it is 
widely implemented. Are you now saying that BGP GR is no good and we 
need something better ? Moreover this something better must be both 
stable and solid for "IP/Routing BGP instance" and flexible and elastic 
to cover every possible hole for "Service BGP" ?

I think if we would address each problem space separately we would be 
much better off with finding the right solution to both.

> after session loss) is wrong.

IMHO the core behaviour of the protocol assumes you have more then one 
path in your BGP table to any important destination. And it is only 
better if those paths are diverse (coming from different control 
planes). Then even if your session goes down immediately nothing bad is 
happening to the system.

If you design your network with a single path only (or same BGP code 
mirrored another one) then sorry it is likely that both sessions go down 
and you loose your service for some time.

In GR you could allow that for 1h .. it is already too long. But to 
allow that for more I am of the opinion that this is not BGP but NETCONF.

> and since BGP has outgrown its

Very good point. If we would start on a per AFI/SAFI basis we would 
perhaps already be at the last call. However authors have chosen not to 
go that path. They have chosen to make BGP persistance for all AFI/SAFIs 
in the spec moreover now they are questioning GR (not advertising routes 
as STALE) and error-handling proposals (which targets to keep the 
sessions up as perhaps malformed attribute is about one prefix out of 
400K).

Best,
R.

---

From: Hannes Gredler <[email-masked]>
Date: Mon, 25 Jun 2012 15:31:10 +0200

On Jun 25, 2012, at 10:26 AM, Robert Raszuk wrote:

> Well actually above I was not intending to sound anti-MPLS. I should say IP/MPLS above .. as you very well know MPLS requires IP to work first !
> 
> It is not about putting services in BGP or not as the protocol. I do not have any problem with that at all. Note that "overload" term was a quote from Jim's mail no my own description.
> 
> It is about running services BGP and IP BGP together which I am questioning. Keyur and I had a draft http://goo.gl/GWrIs "Transport Instance BGP" to allow for clear separation of the two instances.

thats a valid concern - although i have to say that the TI draft is more about implementation and not protocol spec ;-)
wrt to TI -  why restrict to internet vs. non-internet  - to some e.g. the labeled-BGP infra may appear more crucial than the
internet service itself. - i can certainly see the need for a *proper* separation between BGP service instances.

> I do believe that if we would have this in place there would be completely no objection from anyone in IETF to add persistence to "Service BGP". Especially that Bruno mentioned the need to propagate his infrastructure routes in SAFI 1/1. Both BGP instances could operate independently and "IP/Routing instance" could serve very well the "service instance".
> 
> Yes I realize that for you it means more code to maintain so there is price to that. The only question I think which should be stated if perhaps cost vs benefit of such separation is getting to right more now.

unless we have a proper fault containment within any given implementation of BGP,
we can only try to control the failure behavior of routing software, and its garbage collection mechanisms. 

> Hmmmm one would think that RFC4724 addresses that. And moreover it is widely implemented. Are you now saying that BGP GR is no good and we need something better ? Moreover this something better must be both stable and solid for "IP/Routing BGP instance" and flexible and elastic to cover every possible hole for "Service BGP" 

i think extended-GR plus the ability to restart a single AFI/SAFI anytime when the TCP session already has been established
would be more flexible than todays integrated approach where addition/deletion of an AFI/SAFI always restart
in a complete restart of the entire session.

> I think if we would address each problem space separately we would be much better off with finding the right solution to both

i think if you fix BGP implementation and allow graceful restart of sessions
and session instances, then you do not need to worry that much about
the failure behavior and how to mitigate that.

> IMHO the core behaviour of the protocol assumes you have more then one path in your BGP table to any important destination. And it is only better if those paths are diverse (coming from different control planes). Then even if your session goes down immediately nothing bad is happening to the system.

true - multipathing is a viable way of cheating around the problem of control-plane failure ;-)

> If you design your network with a single path only (or same BGP code mirrored another one) then sorry it is likely that both sessions go down and you loose your service for some time

that is understood - 

> In GR you could allow that for 1h .. it is already too long. But to allow that for more I am of the opinion that this is not BGP but NETCONF.

garbage collection intervals of stale prefixes should entirely an operators choice. 
SPs know how large their RIBs are and they know when its time to clean up
stale prefixes;

> Very good point. If we would start on a per AFI/SAFI basis we would perhaps already be at the last call. However authors have chosen not to go that path. They have chosen to make BGP persistance for all AFI/SAFIs in the spec moreover now they are questioning GR (not advertising routes as STALE) and error-handling proposals (which targets to keep the sessions up as perhaps malformed attribute is about one prefix out of 400K).
> 
> Best,
> R.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 25 Jun 2012 15:51:29 +0200

Hi Hannes,

I am glad you fundamentally agree. Just a few follow-up comments ...

> thats a valid concern - although i have to say that the TI draft is
> more about implementation and not protocol spec ;-)

Not really. All it does is clear separation between two BGP instances. 
Sure you can separate them with multisession - what was actually John's 
response in next rev of multisession draft after transport instance got 
posted .. however we went nowhere with both as a result :(

> wrt to TI -  why
> restrict to internet vs. non-internet  - to some e.g. the labeled-BGP
> infra may appear more crucial than the internet service itself. - i
> can certainly see the need for a *proper* separation between BGP
> service instances.

100% agree. In fact I would see labeled-BGP to belong to IP/Routing 
instance .. solid as it can be. That is an infrastructure for a service 
not a service itself.

Besides the TI draft made it very clear that the choice which AFI/SAFI 
is served by which instance would be not be up to IETF but would be left 
to the operator.

The problem with the latter approach of each SAFI having separate 
instance seems to be really difficult to handle by IETF if protocol 
would differ on a per SAFI basis. That's why TI took a very pragmatic 
approach to just allowing number of instances equal 2. But this could be 
up to discussion. Currently we have one or any instance served by the 
same very protocol hence the collisions of group who on one hand want to 
protect the IP/Routing and those on the other hand who careless about IP 
Internet, but do care about their services.

And the most interesting part is that both may be actually correct in 
their own views. Unfortunately single BGP protocol instance causes the 
collision of interests.

>> Yes I realize that for you it means more code to maintain so there
>> is price to that. The only question I think which should be stated
>> if perhaps cost vs benefit of such separation is getting to right
>> more now.
>
> unless we have a proper fault containment within any given
> implementation of BGP, we can only try to control the failure
> behavior of routing software, and its garbage collection mechanisms.

I do agree but that is an implementation. I am just focusing on the 
value of protocol delta between different uses of BGP instances.

> i think extended-GR plus the ability to restart a single AFI/SAFI
> anytime when the TCP session already has been established would be
> more flexible than todays integrated approach where addition/deletion
> of an AFI/SAFI always restart in a complete restart of the entire
> session.

Does not need to. Dynamic capabilities have been invented long time back 
and some are even implementing it :) No longer you need to restart the 
entire session to add or delete a new SAFI.

Ability to restart single AFI/SAFI is again an implementation choice in 
my view.

>> I think if we would address each problem space separately we would
>> be much better off with finding the right solution to both
>
> i think if you fix BGP implementation and allow graceful restart of
> sessions and session instances, then you do not need to worry that
> much about the failure behavior and how to mitigate that.

Maybe ... I am not convinced. Clearly persistent folks are not convinced 
too - otherwise they would not be pushing for it, but instead would be 
banging your door to get new junos version soon :-)

> control-plane failure ;-)

A lot of people consider cheating as a bad thing. So I would not call it 
that way. For me this is rather robust network design.

> garbage collection intervals of stale prefixes should entirely an
> operators choice. SPs know how large their RIBs are and they know
> when its time to clean up stale prefixes;

At this point the spec does not allow that. If STALE is stripped by 
someone there is no base to do any garbage collection any more. Besides 
when each AS would choose to do different time for such garbage 
collection you can imagine what would happen right ?

R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 25 Jun 2012 14:37:27 +0000

Shane

                Comments In-Line..

Jim Uttaro

From: Shane Amante [mailto:[email-masked]]
Sent: Sunday, June 24, 2012 3:05 PM
To: UTTARO, JAMES
Cc: 'Enke Chen'; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

Here's the problem I have with the above.  From the vantage point of my __VPN__ networks (IPVPN, VPLS), I am not observing the same types of network failures (in those networks) as you have observed.  Please note that I am not saying your claims are false, I am only stating that this these types of outages have not occurred in my VPN networks.  As such, I also think it's important for the WG to understand this additional data point from the vantage point of a wholly different set of VPN networks, so that WG members can see a "balanced view".  I don't know (and, likely will never know) why we are not seeing these problem(s) and you are, but I imagine that it's likely due to some combination of, and in no particular order: different HW, different SW and, potentially, different scale.  Ultimately, this operational experience is why I'm stating that this _class_ of problem, specifically related to VPN networks, is very rare.
[Jim U>] That is fair.. AT&T's suite of networks span the globe, We have multiple vendors, numerous releases and multiple platforms.. I cannot divulge actual numbers but a ballpark estimate, exceeds (  10*Internet Scale ) in terms of control plane state. In some case there are individual contexts that rival the global routing table.. I am faced with the dilemma of supporting extremely large networks that support critical applications that simply cannot fail. As Neil pointed out ( Hospital, Police/Fire etc.. and of course Uncle Sam ;) There are challenges that only become evident with large amounts of scale, topologies which are not as robust as we would prefer in different parts of the globe, sheer volume of operations support and possibility of error, different erroneous error i.e mal-formed update etc...

I have read and do support Rob's work.  However, to be perfectly clear, I do believe that we need to solve this overall set of problems (as defined in Rob's draft), just not as has been proposed in the BGP persistence draft.  More to the point, where I __have__ observed this class of problem is related to the Internet side, (AFI/SAFI 1/1 to be more precise), during a handful of occasions over the last several years.  However, most (but, certainly not all) of those failures were clearly attributable to implementation failures (on, the receive side of receiving an UPDATE), for which I don't believe that any solution the IDR WG devises will ever (or, should ever) be able to fix at the end of the day.  So, this ultimately boils down to (again, on the Internet-side) a small number of very large disruptions to the control plane as a result of malformed attributes.  Again, yes, we should solve those.  So, forgive me for being selfish here, but this is why I would rather see the IDR WG solve this in a generic way for all AFI's, so at least if we're going to add complexity to implementations we're _all_ gaining the benefit from it.
[Jim U>] +1 on the support.. Rob's draft should be classified as solving that specific set of use cases identified.. I am not involved on the internet side, but I can definitely see the benefit of solving this for that use case.. I am being selfish also in that I need to solve a bigger set of issues ( See Above ), with different requirements for the other services I am using BGP for..

I'm sorry if you feel that way, but again I am only stating that my experience has been different from yours.  Obviously, each WG member is free to draw their own conclusions as to those individual data points.
[Jim U>] Agreed.. My only point would be that folks do not operate from a perspective of it hasn't happened to me therefore it is not real..

-shane

Jim Uttaro

Enke,

-shane

Jim Uttaro

Jim,

-- Enke

Jim Uttaro

Hi, Saikat:

-- Enke

Robert,

Robert, Randy, Shane,

Senad

Rgs,

R.

Idr mailing list

---

From: t.petch <[email-masked]>
Date: Tue, 26 Jun 2012 16:09:38 +0100

----- Original Message -----
From: "Susan Hares" <[email-masked]>
To: <[email-masked]>
Cc: "'John G. Scudder'" <[email-masked]>
Sent: Friday, June 15, 2012 8:03 PM
> While there is a strong debate on draft-uttaro-idr-bgp-persistence-01
as IDR
> author, I would like to extend the time to consider this draft for 3
more
> To accept the draft we need a few more of you to "hum" yes or "hum"
no.

I do not think that the draft should be adopted in its present form (I
would call it "hum" no but the preceding sentence suggests that that
would be a vote for adoption:-(

It is too broad brush for a change to such a fundamental part of the
Internet as BGP.  I wonder if the consequences have been fully thought
through and believe that the risk is thus too high.

I would support a more tightly focussed I-D, perhaps aimed at a single
SAFI or classified Experimental or ....

In passing, I note that MPLS-TP has a positive requirement that the
forwarding plane stays up when the control plane is recycled so the
underlying idea is reasonable but the consequences need to be clearly
spelled out.

Tom Petch

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 26 Jun 2012 17:32:11 +0000

Hannes,

"the "core behaviour" of the protocol (kill RIB entries right
after session loss) is wrong. and since BGP has outgrown its
original purpose (route 1K prefixes) vs. route 100Ks of prefixes,
route unicast and multicast, route VPN/non-VPN,
route non-routing related functionality it is just legitimate to discuss
restart and persistence behaviour on a per AFI/SAFI basis"

You have exactly captured where I am coming from in re the discussion about persistence. As a standards group we must evolve our thinking to address how BGP is being used today and anticipate its future use. 

Jim Uttaro

-----Original Message-----
From: Hannes Gredler [mailto:[email-masked]] 
Sent: Monday, June 25, 2012 3:08 AM
To: Robert Raszuk
Cc: [email-masked] List; UTTARO, JAMES
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

robert,

 

tx,

/hannes

---

From: Tony Li <[email-masked]>
Date: Tue, 26 Jun 2012 07:47:25 -1000

> You have exactly captured where I am coming from in re the discussion about persistence. As a standards group we must evolve our thinking to address how BGP is being used today and anticipate its f…

Perhaps we should consider the overall routing architecture of the Internet, the role that BGP plays within that architecture, and how major modifications might have implications to the architecture.

Regards,
Tony

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 26 Jun 2012 17:55:15 +0000

Tony,

	I am in no way minimizing the effect of changing a base assumption in re persistence as it applies to the internet although it seems that the IETF community has accepted GR which is also creates a persistence paradigm for a smaller amount of time.. As stated in the BGP Persistence draft IPV4 is not a candidate use case, but there are other use cases which require it. 

How do you think we should proceed? Should the draft explicitly state that the internet use case is a MUST NOT? The 3107 use case is labeled IPv4 which has persistence requirements also so I do not know how that could actually be done..

Jim Uttaro

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tony Li
Sent: Tuesday, June 26, 2012 1:47 PM
To: UTTARO, JAMES
Cc: Robert Raszuk; [email-masked] List
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Regards,
Tony

---

From: Brian Dickson <[email-masked]>
Date: Tue, 26 Jun 2012 16:20:15 -0400

I am not in favor of adoption - "hum" == no. I am sympathetic to the
author(s), however.

There are places where state information can be traded off for stability
and performance, but this isn't it, IMHO.

(Also, I hope at some point to revisit stuff from a few years ago, work on
improving stability/performance/convergence,
at the cost of modest amount of state, in the near future, but no promises
just yet.)

And Tony is, as usual, spot on.

Brian
(And yes, I've operated very big networks with lots of prefixes, and have a
ton of deep BGP experience. Not that we're measuring anything.)

(FWIW, I'd be happy to assist, Jim, in exchange for beer(s), if you're in
N. VA area. Contact me off-list.)

On Tue, Jun 26, 2012 at 1:47 PM, Tony Li <[email-masked]> wrote:

> > You have exactly captured where I am coming from in re the discussion
> about persistence. As a standards group we must evolve our thinking to
> address how BGP is being used today and anticipate its future use.
>
>
> Perhaps we should consider the overall routing architecture of the
> Internet, the role that BGP plays within that architecture, and how major
> modifications might have implications to the architecture.
>
> Regards,
> Tony
>

---

From: "Simpson, Adam (Adam)" <[email-masked]>
Date: Tue, 26 Jun 2012 16:10:35 -0500

I have been following the debate on this draft fairly carefully, and even though I am one of the authors, I have tried to be as open-minded as possible to the different points of view. I can see that some of the text in draft-01 is too loose - particularly around limiting the  advertisement scope of the Stale community and deciding what to do at the boundaries of the persistence domain, interaction with the approaches described in draft-ietf-idr-error-handling, and minimizing churn when there are repeated resets of a "persistence-enabled" session. But Bruno, Jim and John have already started to indicate how a draft-02 version could/would address these points and in my mind those changes would alleviate a lot of the concerns. I don't want to enter the debate about how unlikely a catastrophic control plane failure (decoupled from forwarding) is relative to other 'rare' events such as malformed Update packets, etc. because I know that no two networks are exactly the same and I know that just because something hasn't happened to me yet doesn't mean it can't happen to me in the future. Finally - to those that have argued that adding BGP persistence into router software code has the potential to introduce new bugs and other unintended consequences I won't deny this is a possibility but from an implementer's perspective BGP persistence appears to be a much simpler and straightforward code change (conceptually it has shared elements with GR and graceful shutdown) than say the code changes implied by draft-ietf-idr-error-handling. This is not to argue for one vs. the other - I think they can be complementary - but I think this perspective is important.

-Adam

IDR WG and BGPers:

Sue Hares

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 26 Jun 2012 17:44:26 -0400

To address the problem of retaining BGP routes after
failure of the BGP session, I am preparing a draft
that starts like this:

   When a BGP session terminates, the standard response is to delete all
   information learnt from that session. Even when a BGP session fails,
   the routes learnt from it may still be viable. To maintain network
   connectivity, it is desirable to retain routes as long as their
   viability can be assured.

   If the next-hop is reachable, then we can send it a message to ask.

I can understand the desire to keep STALE routes.
The objection I see is that the viability of a STALE route is not assured.
If a simple enough protocol can be developed to verify route viability,
it would provide extra confidence for STALE routes.

Is there interest to develop this?

-- 
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 27 Jun 2012 00:06:55 +0200

Hi Jakob,

I think this is a basic assumption even for the current persistence 
draft version that next hop is reachable in the RIB.

The draft fails to discuss what happens if we are dealing with double or 
N-level BGP recursion though.

However let me point out that next hop reachability from control plane 
RR point of view - which is not in the data path - does not really mean 
that all of it's clients would have an identical view.

For that there is existing draft already proposed which allows RR to get 
next hop cost and reachability information from it's clients: 
http://goo.gl/jQfnB "Carrying next-hop cost information in BGP"

Perhaps it could be reused.

Thx,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 26 Jun 2012 18:10:23 -0400

This is not about next-hop reachability.
It is the next step.
Ask the nexthop if the route is viable.
Not with BGP. This is for use when BGP is dead.

On Tuesday, June 26, 2012 3:07 PM, Robert Raszuk <mailto:[email-masked]> wrote:

>>     When a BGP session terminates, the standard response is to
>>     delete all information learnt from that session. Even when a BGP
>>     session fails, the routes learnt from it may still be viable. To
>>     maintain network connectivity, it is desirable to retain routes
>> as long as their     viability can be assured. 
>> 
>> The objection I see is that the viability of a STALE route is not
>> assured. If a simple enough protocol can be developed to verify
>> route viability, it would provide extra confidence for STALE routes.
>> 

-- 
Jakob Heitz. x25475. 510-566-2901

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 27 Jun 2012 00:17:37 +0200

> Not with BGP. This is for use when BGP is dead.

You mean when BGP session(s) or process is dead you will query all next 
hops if all routes you have received are still their best ?

Is that the plan ?

R.

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 26 Jun 2012 18:23:36 -0400

Say I have a STALE route: destination 10.0.0.0/24 with next-hop 10.1.1.666.
I would send a message to 10.1.1.666 asking it if it has a route to 10.0.0.0/24.

Of course, there are issues.
If we can work through the issues,
is there interest?

-- 
Jakob Heitz.

---

From: "Senad .Palislamovic" <[email-masked]>
Date: Tue, 26 Jun 2012 18:24:47 -0400

Robert, Enke,

Thanks for the inputs.  Your concern with Internet routes is valid one.

Shane,

Would you agree that concern of introducing "bugs" with this change is more
less the same as any feature being introduced into BGP?

And yet, where there are business cases and it just makes technological
sense to go down that path, we all agree and proceed.  Thus we have reached
the point that we have numerous new features and services in BGP showing up
as some sort of route (for forwarding or topology discovery purposes) -
PIM/IGMP state in BGP in MVPNs (6513/14), MAC state (evpn draft), VPLS -
4761, BGP-TE draft.  My point is that we don't stop innovation because of
potential bugs - it is up to the vendor to ensure bug-free (dream) code.
 In this case, there are several operators who do have a business case for
this feature, so why stop now?

Having said that, I fully understand and agree with Robert's concern of
implementing this on pure v4/v6 Internet routes - having two diverse paths
on my network, I might not ever want to take "STALE" route on someone
else's network - 1-2 ASs down the road - so some sort of "consistency"
across the entire Internet on messaging the "STALE" info is needed.  It
seems that both Enke and Saikat have proposed two possible solutions for
Internet space - either capability advertisement OR "STALE" community to
stay with the route indefinitely, ie, CANNOT be removed.

With that said, apart from Internet routes, I fail  to see any
technological or philosophical issue in going forward with BGP persistance
and having operators enable it on their services as they see it fit.

Senad

---

From: Tony Li <[email-masked]>
Date: Tue, 26 Jun 2012 16:24:29 -1000

> How do you think we should proceed? Should the draft explicitly state that the internet use case is a MUST NOT? The 3107 use case is labeled IPv4 which has persistence requirements also so I do not…

Aloha James,

We should proceed as always.  Those who believe that this problem should be solved should work with those who are willing to solve the problem and present a solution to the group.  Preferably with running code.

As always, the burden falls to those proposing the changes to convince the rest of us that the benefits of the proposal exceed its drawbacks.

Mahalo,
Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 26 Jun 2012 16:29:45 -1000

Aloha,

On Jun 26, 2012, at 12:24 PM, Senad .Palislamovic wrote:

> Would you agree that concern of introducing "bugs" with this change is more less the same as any feature being introduced into BGP?

I would agree that that is always a concern, as with any change.  I would also agree that the risk for bugs with this proposal is high, and that the number of applicable use cases seems to be small.

> And yet, where there are business cases and it just makes technological sense to go down that path, we all agree and proceed.  

Sorry, but violently, no.  We agreed because we had rough consensus, NOT because there is a business case.  There are many considerations beyond a business case and 'technological sense'.  This includes architectural direction and increased complexity.  These must be weighed in making a rational decision.  Hopefully all parties are trying to make a _rational_ decision.

> Thus we have reached the point that we have numerous new features and services in BGP showing up as some sort of route (for forwarding or topology discovery purposes) - PIM/IGMP state in BGP in MVPNs (6513/14), MAC state (evpn draft), VPLS - 4761, BGP-TE draft.  My point is that we don't stop innovation because of potential bugs - it is up to the vendor to ensure bug-free (dream) code.  In this case, there are several operators who do have a business case for this feature, so why stop now?

Not everyone agrees that the aforementioned features should have been included in the first place.  Again, it requires rough consensus to move forward and there are those that will dissent.

Mahalo,
Tony

---

From: Shane Amante <[email-masked]>
Date: Tue, 26 Jun 2012 20:43:19 -0600

Senad,

On Jun 26, 2012, at 4:24 PM, Senad .Palislamovic wrote:
> And yet, where there are business cases and it just makes technological sense to go down that path, we all agree and proceed.  Thus we have reached the point that we have numerous new features and services in BGP showing up as some sort of route (for forwarding or topology discovery purposes) - PIM/IGMP state in BGP in MVPNs (6513/14), MAC state (evpn draft), VPLS - 4761, BGP-TE draft.  My point is that we don't stop innovation because of potential bugs - it is up to the vendor to ensure bug-free (dream) code.  In this case, there are several operators who do have a business case for this feature, so why stop now?

I find it disingenuous, to put it politely, that "it just makes technological sense to go down that path".  Since you brought it up by citing a wide variety of additional features that have been added, or are still being [seriously] considered being added to BGP, where do we draw the line?  No, /seriously/, I **really** want to know.  To put it bluntly, it seems there's NOT really a feature that BGP hasn't been adapted to support.  Or, to put it more colloquially: "If BGP is a hammer, [nearly] every problem is a nail".

On the one hand, that's a marvelous testament to the original design of BGP to be able to "easily" adapt to support new services, allowing for moderately fast roll-out on SP networks.  However, when looked at from the context of actually operating networks that run BGP, i.e.:
- draft-ietf-grow-ops-reqs-for-bgp-error-handling;
- my own and others studies into the present scalability of iBGP;
- lots of research papers documenting various aspects of BGP, (Labovitz/Ahuja; Route Flap Damping; studies on CLUSTER_LIST changes, but not their length, causing flooding of new iBGP updates inside and outside an ASN, etc.)
... I am very concerned that if we keep /incrementally/ pouring new features into BGP without considering the overall effects on BGP as a _system_ then we're not doing a good enough job.  By a "system" I mean of networks consisting of not just 1 router in an ASN, but thousands of routers in a single [VPN] network, and orders of magnitude more in the DFZ.  FWIW, that is the context by which I evaluate drafts proposed in this, and other, WG's.

Furthermore, and back to the discussion of this particular draft, I acknowledge that BGP session survivability is a problem; however, in the context of the networks that I operate:
a)  It has not occurred in my VPN networks;
b)  It has occurred, but only very rarely in my Internet network, (and even then several of those events are clearly attributable to _implementation_ errors that nothing we do in IDR will ever fix).

Ultimately, I really do want the IDR WG to solve this problem, but I hope that we will seriously raise the bar in terms of what we, as operators, are expecting for a solution here.  IMO, I think that is the spirit (if not more) of why draft-ietf-grow-ops-reqs-for-bgp-error-handling was written.  Personally, at this stage (and given the data I cite above wrt my own networks), I believe we have a moderate amount of time that would allow us to raise the bar on our expectations.  And, I would rather set the bar too high, and miss it, then only have a partial solution that is the currently proposed persistence draft.  IMHO, I would much rather see a solution that:
a) has consistent support across _ALL_ AFI/SAFI's -- that benefits everyone.  It allows for a consistent training regime for my Ops folks, regardless of which services or networks they are working on, etc.
b) takes a much more "optimistic" approach to dynamically 'fixing' the problem in the control plane.  Namely, it's my belief (and, I wish I had hard data to back this up), that it's only an extremely small handful of BGP routes with malformed attributes that are the problem here, (in reality ... in only takes 1 malformed attribute to cause chaos, right?)  If that's the case, I would much rather see a solution that tries to contain the small number "bad" routes in a type of 'penalty box' and /NOT/ disturb the overwhelming majority of the existing "good" routes, at all.
  c) IOW, there's no need for me, as an operator, to pre-color routes with communities that so those routes can survive session failures.  (WRT to persistence, I will /always/ guess wrong which customer's routes to enable it on and which don't care, so at the end of the day operators are going to turn it on for ALL routes).
  d) In addition, when a BGP receiver gets "choked up", his neighbors aren't required to re-flood out routes with a new community on them indicating something 'bad' has happened.  IOW, the problem should be contained as locally as possible in the AS, w/out flooding this throughout the whole ASN -- if possible/practical.  My routers are busy enough handling flapping PE-CE circuits and dealing with churn from all the existing wonderful "features", that I really don't need addt'l churn /if/ it can be avoided in the first place.

At the scale of BGP that we're operating in our networks today, this is not a time for half-measures to this problem.  We should aim to fix it once and for all.  /If/ we aim to do that, then I'm all for it.

My apologies if the above sounds too much like a "rant" ... but, it's been a long day.  :-)

As always, just my $0.02,

-shane

> Having said that, I fully understand and agree with Robert's concern of implementing this on pure v4/v6 Internet routes - having two diverse paths on my network, I might not ever want to take "STALE" route on someone else's network - 1-2 ASs down the road - so some sort of "consistency" across the entire Internet on messaging the "STALE" info is needed.  It seems that both Enke and Saikat have proposed two possible solutions for Internet space - either capability advertisement OR "STALE" community to stay with the route indefinitely, ie, CANNOT be removed.  
> 
> With that said, apart from Internet routes, I fail  to see any technological or philosophical issue in going forward with BGP persistance and having operators enable it on their services as they see it fit.
> 
> Senad
> 
> 
> 
>

---

From: "Senad .Palislamovic" <[email-masked]>
Date: Wed, 27 Jun 2012 01:18:31 -0400

Tony,

My comments inline:

On Tue, Jun 26, 2012 at 10:29 PM, Tony Li <[email-masked]> wrote:

> > Would you agree that concern of introducing "bugs" with this change is
> more less the same as any feature being introduced into BGP?
>
>
> I would agree that that is always a concern, as with any change.  I would
> also agree that the risk for bugs with this proposal is high, and that the
> number of applicable use cases seems to be small.
>
> I do not think that "number" is the right unit of measure here.  Number is
only relevant as much as the monetary system it is tied to.  In this case,
a significance of the use cases, IMHO outweighs the risk.

> sense to go down that path, we all agree and proceed.
>
>
> Sorry, but violently, no.  We agreed because we had rough consensus, NOT
> because there is a business case.

Could it be that a business case drove the "rough consensus"?  And didn't
insertion of these services into BGP made operators realize new revenue
streams - which didn't exist before - sooner.  Thus the adaptation rate.

 There are many considerations beyond a business case and 'technological
> sense'.  This includes architectural direction and increased complexity.
>  These must be weighed in making a rational decision.  Hopefully all
> parties are trying to make a _rational_ decision.
>
> Fully agree.  However, different services call for different architectural
directions - thus their level of complexity will be different.  I view
complexity as a relative term - complex compared to what ?

> > Thus we have reached the point that we have numerous new features and
> services in BGP showing up as some sort of route (for forwarding or
> topology discovery purposes) - PIM/IGMP state in BGP in MVPNs (6513/14),
> MAC state (evpn draft), VPLS - 4761, BGP-TE draft.  My point is that we
> don't stop innovation because of potential bugs - it is up to the vendor to
> ensure bug-free (dream) code.  In this case, there are several operators
> who do have a business case for this feature, so why stop now?
>
>
> Not everyone agrees that the aforementioned features should have been
> included in the first place.  Again, it requires rough consensus to move
> forward and there are those that will dissent.

Fair enough.  However, given that we have moved forward, implemented it,
deployed our services, locked the customers and revenue streams in -
shouldn't we seek to improve them?

Regards,

Senad

---

From: "Senad .Palislamovic" <[email-masked]>
Date: Wed, 27 Jun 2012 02:12:41 -0400

Shane,

I sincerely appreciate your 2 cents.  My comments inline:

On Tue, Jun 26, 2012 at 10:43 PM, Shane Amante <[email-masked]>wrote:

> I find it disingenuous, to put it politely, that "it just makes
> technological sense to go down that path".  Since you brought it up by
> citing a wide variety of additional features that have been added, or are
> still being [seriously] considered being added to BGP, where do we draw the
> line?  No, /seriously/, I **really** want to know.  To put it bluntly, it
> seems there's NOT really a feature that BGP hasn't been adapted to support.
>  Or, to put it more colloquially: "If BGP is a hammer, [nearly] every
> problem is a nail".
>

The right question to ask would be why to draw the line at the first place.
 Doesn't leveraging and re-using something that exists and is proven to
work make more sense than going down the path of reinventing the wheel - as
long as the following factors are met:

- it doesn't impact the original product
- the original product contains the right properties to be extended

Has any of the new services "broke" the original product,
ie vanilla routing?  Hybrid vehicles use roughly 95 % components of their
non-hybrid brothers.  Car industry didn't go out and build the hybrid
vehicle in the vacuum, but actually re-used existing technologies as much
as possible.

> On the one hand, that's a marvelous testament to the original design of
> BGP to be able to "easily" adapt to support new services, allowing for
> moderately fast roll-out on SP networks.  However, when looked at from the
> context of actually operating networks that run BGP, i.e.:
> - lots of research papers documenting various aspects of BGP,
> (Labovitz/Ahuja; Route Flap Damping; studies on CLUSTER_LIST changes, but
> not their length, causing flooding of new iBGP updates inside and outside
> an ASN, etc.)
> ... I am very concerned that if we keep /incrementally/ pouring new
> features into BGP without considering the overall effects on BGP as a
> _system_ then we're not doing a good enough job.  By a "system" I mean of
> networks consisting of not just 1 router in an ASN, but thousands of
> routers in a single [VPN] network, and orders of magnitude more in the DFZ.
>  FWIW, that is the context by which I evaluate drafts proposed in this, and
> other, WG's.
>

So the question is - why should it be looked as a whole system.  What
defines a system?  Everything is fundamentally layered on something - so
leverage the commons and diverge where needed.  Any protocol stack is built
on same fundamental layer and diverges as it moves up based on its
requirements and problem statements.  In this case, different NLRI's share
different properties - so we re-use what is common and adapt where we
diverge according to the properties of the services.  I think we have moved
passed vanilla IP services long time ago.

> Furthermore, and back to the discussion of this particular draft, I
> acknowledge that BGP session survivability is a problem; however, in the
> context of the networks that I operate:
> b)  It has occurred, but only very rarely in my Internet network, (and
> even then several of those events are clearly attributable to
> _implementation_ errors that nothing we do in IDR will ever fix).
>

> Ultimately, I really do want the IDR WG to solve this problem, but I hope
> that we will seriously raise the bar in terms of what we, as operators, are
> expecting for a solution here.  IMO, I think that is the spirit (if not
> more) of why draft-ietf-grow-ops-reqs-for-bgp-error-handling was written.
>

I do not think the draft draft-ietf-grow-ops-reqs-for-bgp-error-handling
addresses this problem.

Personally, at this stage (and given the data I cite above wrt my own
> networks), I believe we have a moderate amount of time that would allow us
> to raise the bar on our expectations.  And, I would rather set the bar too
> high, and miss it, then only have a partial solution that is the currently
> proposed persistence draft.  IMHO, I would much rather see a solution that:
> a) has consistent support across _ALL_ AFI/SAFI's -- that benefits
> everyone.  It allows for a consistent training regime for my Ops folks,
> regardless of which services or networks they are working on, etc.
>

Given that each AFI/SAFI set defines a service with potentially
fundamentally different properties, I do not think one size fits all
methodology is appropriate here.  Going back to my previous comment - use
and re-use on things that are common.

Now let's look at very specific use case, RFC 4761.  BGP (RRs) are used for
auto-discovery and PW instantiation.  Once that is up, all  forwarding
state is after-fact of dynamic MAC learning within a pseudowire.  Given
that provider topology is static, BGP can be removed from the picture,
moreover RR can be removed from the picture.  This absolutely doesn't
impact customer traffic cuz destinations are based on dynamic MAC learning
so any farther changes in topology will be reflected on the customer's
network correctly.

This is fundamentally different use case than what you are concerned with,
and IMHO could leverage different set of solutions, as long as they
"fundamentally" do not impact the original product (IDR).

> b) takes a much more "optimistic" approach to dynamically 'fixing' the
> problem in the control plane.  Namely, it's my belief (and, I wish I had
> hard data to back this up), that it's only an extremely small handful of
> BGP routes with malformed attributes that are the problem here, (in reality
> ... in only takes 1 malformed attribute to cause chaos, right?)  If that's
> the case, I would much rather see a solution that tries to contain the
> small number "bad" routes in a type of 'penalty box' and /NOT/ disturb the
> overwhelming majority of the existing "good" routes, at all.
>   c) IOW, there's no need for me, as an operator, to pre-color routes with
> communities that so those routes can survive session failures.  (WRT to
> persistence, I will /always/ guess wrong which customer's routes to enable
> it on and which don't care, so at the end of the day operators are going to
> turn it on for ALL routes).
>   d) In addition, when a BGP receiver gets "choked up", his neighbors
> aren't required to re-flood out routes with a new community on them
> indicating something 'bad' has happened.  IOW, the problem should be
> contained as locally as possible in the AS, w/out flooding this throughout
> the whole ASN -- if possible/practical.  My routers are busy enough
> handling flapping PE-CE circuits and dealing with churn from all the
> existing wonderful "features", that I really don't need addt'l churn /if/
> it can be avoided in the first place.
>

This would only happen in v4/v6 Internet routing space - not in SP owned
services or infrastructure routes.  This is why I believe different NLRIs,
given that they have different set of properties, could leverage different
solution.

> At the scale of BGP that we're operating in our networks today, this is
> not a time for half-measures to this problem.  We should aim to fix it once
> and for all.  /If/ we aim to do that, then I'm all for it.
>
No worries.  it's only Tuesday - wait till it gets to be Thursday :)

> As always, just my $0.02,
Likewise, just MHO...

Senad

---

From: Tony Li <[email-masked]>
Date: Tue, 26 Jun 2012 21:42:16 -1000

Aloha Senad,

> I do not think that "number" is the right unit of measure here.  Number is only relevant as much as the monetary system it is tied to.  In this case, a significance of the use cases, IMHO outweighs the risk.

Number is relevant as the group is driven by rough consensus.  A particular issue can have arbitrarily high significance to some, but if there is no rough consensus within the group, then the proposal will not progress.

>> Sorry, but violently, no.  We agreed because we had rough consensus, NOT because there is a business case.
>> 
> Could it be that a business case drove the "rough consensus"?  And didn't insertion of these services into BGP made operators realize new revenue streams - which didn't exist before - sooner.  Thus the adaptation rate.

Perhaps.  It's not clear what the motivations are from all participants.

>>  There are many considerations beyond a business case and 'technological sense'.  This includes architectural direction and increased complexity.  These must be weighed in making a rational decision.  Hopefully all parties are trying to make a _rational_ decision.
>> 
> Fully agree.  However, different services call for different architectural directions - thus their level of complexity will be different.  I view complexity as a relative term - complex compared to what ?

Interesting.  As a computer scientist, I view complexity as both a quantifiable issue (i.e., computational and space complexity) and as a pragmatic issue both for developers and operators.  

Sure, an algorithm may run in O(1) time.  All wonderful, but if the logic involved is so tortured that the developers cannot get it right, if the testers can't tell when it is and is not working correctly, and when it causes more operational issues than it solves, then the proposal has become too complex.

An extremely important part of our job is to judge exactly that complexity as part of any proposal that is put forth.

> Fair enough.  However, given that we have moved forward, implemented it, deployed our services, locked the customers and revenue streams in - shouldn't we seek to improve them?

A fine goal to be sure, but improvements come with costs.  Those must be considered.

Mahalo,
Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 26 Jun 2012 21:58:51 -1000

[Removed ietf-action … don't know why they were on this thread…]

On Jun 26, 2012, at 8:12 PM, Senad .Palislamovic wrote:

> The right question to ask would be why to draw the line at the first place.  Doesn't leveraging and re-using something that exists and is proven to work make more sense than going down the path of reinventing the wheel - as long as the following factors are met:
> 

Fact: it  impacts the original product.  If you add a subsystem to an existing system, even if the new subsystem is unused, you've had an impact.  If nothing else, you've added complexity.  More likely, you've added interactions between subsystems.  These are not free.

Leveraging and re-using certainly have their benefits, but they also have their drawbacks.  Sometimes it is the architect's job to step back, throw away the original infrastructure and start afresh.

You should be very happy about this, otherwise we would be having a discussion about extending GGP or EGP instead.  ;-)

Sometimes, it makes sense to start wholly independent subsystems.  It's a judgement call that we must make together.

> Has any of the new services "broke" the original product, ie vanilla routing?  

Yes, absolutely.  We have seen numerous bugs introduced into vanilla routing because of implementation issues of new functions.  For example, we had several issues around capability negotiations.

> So the question is - why should it be looked as a whole system.  What defines a system? 

Based on your LinkedIn resume, I suspect you know this answer:  a system is a set of interconnected subsystems.  BGP should be looked at as a system because when a particular network element becomes a BGP speaker, it normally ends up with a single implementation support multiple AFI/SAFI.  Thus, it all ties into a single, global system.  

Mahalo,
Tony

---

From: Warren Kumari <[email-masked]>
Date: Wed, 27 Jun 2012 12:34:43 +0200

> I do not think that "number" is the right unit of measure here.  Number is only relevant as much as the monetary system it is tied to.  In this case, a significance of the use cases, IMHO outweighs…

Wow… just wow…

W

--
"Let's just say that if complete and utter chaos was lightning, he'd be the sort to stand on a hilltop in a thunderstorm wearing wet copper armour and shouting 'All gods are bastards'."

    -- Rincewind discussing Twoflower (Terry Pratchett, The Colour of Magic)

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 27 Jun 2012 18:18:51 +0000

All,

                After reviewing all of the comments in re BGP Persistence I would categorize the opinions of the WG ( At least the ones that have chimed in )  in the following manner. I would like to get back to a discussion about the facts..

a) Folks would believe that the BGP should never have been used for any other services except for the one originally envisaged..

These opinions suggest that if we simply had not extended BGP to other use cases we would not have to now deal with new/different/addl requirements that these services demand.. Some have stated or implied  that we simply do not extend BGP further, roll back BGP to only Internet application, and/or convince ourselves that the requirements for these new services are not different than the original internet use case.

IMO this discussion is not appropriate to BGP Persistence.. This discussion has been settled, the WG has adopted drafts that extend BGP to other use cases.  Implicit in this adoption is an acknowledgment that these services have different requirements than the original use case and addl drafts/technical specifications  have to be created to meet these needs. BGP persistence is an example of this..

I do think that drafts of new technical specifications may not have been complete in defining the semantic or requirements.. As an ex, the authors of flowspec envisaged a use of BGP to create what is essentially configuration of a traffic filter.. Generally speaking, configuration is static and persists through session failures. That same paradigm applies to flowspec.

b) Folks who feel it is unacceptable that BGP Persistence creates STALE state in domains adjacent to the persistent domain.

Two points here..

The first is that this is the current behavior as defined by GR for the internet use case with the fact that there is no method of GR informing that paths are STALE. I have repeatedly noted this and so far E. Chen is the only person that acknowledged the behavior with a caveat that the behavior is for a short amount of time and therefore acceptable.. I have heard no other opinions on this are there any?

The second is BGP Persistence draft took note of the above fact in re GR and created the ability to inform topologies by th setting of a STALE CV.. Adjacent domains can easily identify the paths and decide on whether or not they should be used. So the draft directly addresses this issue.

So IMO

(a) concerns are of the religious variety.. I cannot speak to the consensus of the WG when addl use cases for BGP were adopted. But this is the reality of where we are.. If IDR is going to be the venue for these discussions then we need to have a consistent view of the reality of BGP and its use cases today and into the future. If not, then I would respectfully submit that the newer use cases be split off from the internet use case and handled in a different venue.

(b) This I truly do not understand.. My co-authors and I have provided a simple mechanism to create a persistent solution and included and ability to inform peer topologies as this IMO seemed like a hole in the current GR draft for this functionality. We have also specifically stated that internet is not a use case..

Jim Uttaro

From: [email-masked] [mailto:[email-masked]] On Behalf Of Shane Amante
Sent: Tuesday, June 26, 2012 10:43 PM
To: Senad .Palislamovic
Cc: [email-masked] List; UTTARO, JAMES; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Senad,

Shane,

-shane

Senad

On Tue, Jun 26, 2012 at 1:32 PM, UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> wrote:
Hannes,

Jim Uttaro

-----Original Message-----
From: Hannes Gredler [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, June 25, 2012 3:08 AM
To: Robert Raszuk
Cc: [email-masked]<mailto:[email-masked]> List; UTTARO, JAMES
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks
robert,

tx,

/hannes

---

From: Shane Amante <[email-masked]>
Date: Wed, 27 Jun 2012 13:35:45 -0600

Jim,

>                 After reviewing all of the comments in re BGP Persistence I would categorize the opinions of the WG ( At least the ones that have chimed in )  in the following manner. I would like…

[--snip--]

> (b) This I truly do not understand.. My co-authors and I have provided a simple mechanism to create a persistent solution and included and ability to inform peer topologies as this IMO seemed like…

I have significant concerns wrt (b), in particular your last sentence above.  More specifically, this is the IDR WG, where the charter currently states:
---snip---
    The main objective of the working group is to support the use of
    BGP-4 by IP version 4 and IP version 6 networks. The working group
    will also continue to work on improving the robustness and
    scalability of BGP.
---snip---
IMHO, both sentences appear to state that the WG should focus on developing the robustness and scalability of BGP for all types of networks, not just VPN networks.  As such, it may be prudent for the WG to consider acknowledging this is a problem with the _architecture_ of BGP, applicable to all AFI/SAFI's, and diligently work toward a single solution that addresses this problem in a comprehensive and consistent manner across all AFI/SAFI's.

-shane

---

From: Brian Dickson <[email-masked]>
Date: Wed, 27 Jun 2012 17:05:45 -0400

On Wed, Jun 27, 2012 at 2:18 PM, UTTARO, JAMES <[email-masked]> wrote:

>  All,****
>
> ** **
>
>                 After reviewing all of the comments in re BGP Persistence
> I would categorize the opinions of the WG ( At least the ones that have
> chimed in )  in the following manner. I would like to get back to a
> discussion about the facts..
>
> ** **
>
> b) Folks who feel it is unacceptable that BGP Persistence creates STALE
> state in domains adjacent to the persistent domain.****
>
> ** **
>
> Two points here.. ****
>
> ** **
>
> The first is that this is the current behavior as defined by GR for the
> internet use case with the fact that there is no method of GR informing
> that paths are STALE. I have repeatedly noted this and so far E. Chen is
> the only person that acknowledged the behavior with a caveat that the
> behavior is for a short amount of time and therefore acceptable.. I have
> heard no other opinions on this are there any? ****
>
> ** **
>
> The second is BGP Persistence draft took note of the above fact in re GR
> and created the ability to inform topologies by th setting of a STALE CV..
> Adjacent domains can easily identify the paths and decide on whether or not
> they should be used. So the draft directly addresses this issue.****
>
> ** **
>
> So IMO
>
> ** **
>
> (b) This I truly do not understand.. My co-authors and I have provided a
> simple mechanism to create a persistent solution and included and ability
> to inform peer topologies as this IMO seemed like a hole in the current GR
> draft for this functionality. We have also specifically stated that
> internet is not a use case..
>

I have a few questions, after more carefully reading the -01 draft.

Please indulge me, I am interested in understanding the relationship
between the motivation/goal, the specific use cases presented, and the
mechanisms themselves.

First, it seems that the "STALE" stuff is partly motivated by a perceived
lack of notification in GR. Isn't that irrelevant to the draft at hand, and
perhaps better addressed by updating GR?

Also, in the use cases, there does not seem to be any real value to
advertising this "STALE" state beyond the local systems. In the L2VPN in
particular, there is no "beyond" at all. So, why bother with "STALE" at
all? I don't see it being used anywhere in your draft, only that it is
being set.

The main problem (that needs to be solved, i.e. the motivation for
Persistence) seems to be that intra-AS PE routers, doing L2VPN, would like
to maintain state in the case where one or more intermediate RRs "go away"
for some period of time.
The scope of "state" information, is more closely aligned with the BGP peer
state, rather than attributes of individual NLRIs of any given AFI/SAFI.

Would it not be (a) simpler, (b) more scalable, and (c) more trivial to
implement, by creating an Internet-Draft for a negotiated option,
"Persistent", where, when two or more parties negotiate this option, and
NLRIs heard over such sessions being "held" if/when a BGP session is
terminated?

This would not have any direct impact to path selection, for example, and
would fairly trivially support negotiated (based on local configuration of)
timers and "reaping" NLRIs upon timer expiry. Behavior per-AFI/SAFI should
be able to be specified.

Not needing to change the content or format of UPDATEs should be seen as a
goal, rather than the other way round.

Let alone further overloading the COMMUNITY attribute, with mandated
behavior based on something that is both optional _and_ transitive. (Yuck.)

In fact, would not this fit nicely as a sub-code to GR, with a negotiated
status and timer between peers (or RR servers/clients)?

Perhaps having the "negotiation" have multiple values, such as "never",
"allow", "prefer", "always", so that in appropriately configured networks,
on a per-peer (or peer-group) basis, it could be centrally controlled by
the RR primarily.

One other question I have, is about Section 6.1.
I do not understand a couple of things about the diagram and text:
Is the diagram just messed up a bit, with RR2 meaning to talk to PE1 and
PE2, rather than RR2 talking to  RR1 and CE1?

Which BGP sessions are presumed to go away, i.e. where Persistence will
make a difference (or fix a problem you're seeing)?
Is it the case that *both* PE1-RR1 *and* PE1-RR2 BGP sessions fail
simultaneously?

And now to dig deeper into the diagram:
Are CE1-PE1 and CE2-PE2 connections over switched (L2 Ethernet) connections?

Would the problem you are experiencing not be possible to fix with, say,
PVRST over dual-PE1's facing CE1, and similarly dual-PE2's facing CE2, in a
redundant configuration? That way, only PE1a (and not PE1b) would dump BGP
peering, and things would fail-over to PE1b? (At which point it is a
hardware cost and network design exercise, rather than a protocol thing.)

You'd have two wires (PE1a-PE1b, and PE2a-PE2b) and two pseudo-wires
(PE1a-PE2a, ,and PE1b-PE2b) forming a "square" topology on which some
spanning-tree thing would need to turn into a loop-free topology, and it
wouldn't matter which PE1 or PE2 was preferred by CE1 and CE2 respectively.

Sincerely,
Brian Dickson

---

From: John G.Scudder <[email-masked]>
Date: Thu, 28 Jun 2012 15:30:52 -0400

Jakob,

> unless it is the default route or also SUSPECT.

If by "will forward packets" you mean "... to their intended destination" this is not generally correct. Consider the use of more-specifics for prefix portability from one provider to another. The provider that supplied the prefix continues to own and advertise the supernet, but no longer provides service to their dear departed customer.

> set of ALL available paths, including the less specific ones.

Nope, see above.

--John

---

From: John G.Scudder <[email-masked]>
Date: Thu, 28 Jun 2012 15:30:59 -0400

On Jun 22, 2012, at 9:07 AM, Robert Raszuk wrote:

> errors which would trigger it down are received. 

None of these address a problem at or below the transport level, though. The conversation so far has largely neglected this point.

--John

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 28 Jun 2012 22:52:55 +0200

Hi John,

Consider GR.

If on the restarting speaker BGP sessions go down due to TCP bug or 
TCP-BGP interaction bug wouldn't receiving speaker apply BGP GR procedure ?

R.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Thu, 28 Jun 2012 23:58:21 +0000

Shane,

---snip---

Jim U> I believe that different use cases require different behaviors. There is simply no way of getting around this.. As I stated in my ex.. Flowspec is a method of creating configuration via BGP.. This should persist as if it was programmed via XML, etc...  I always want robustness and scalability but certain AFs require more than this. I think the charter should to be amended to accomodate the different use cases for BGP and where they diverge in terms of behavior, customer expectations and solutions, or at minimum try to operate from a framework that acknowledges these use cases and their different requirements.

Jim Uttaro

From: Shane Amante [mailto:[email-masked]]
Sent: Wednesday, June 27, 2012 3:36 PM
To: UTTARO, JAMES
Cc: Senad .Palislamovic; [email-masked]<mailto:[email-masked]> List; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

[--snip--]

Two points here..

So IMO

---snip---

-shane

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 29 Jun 2012 01:36:42 +0000

Brian,

                Appreciate your comments. Comments In-Line..

Thanks,
Jim Uttaro

From: Brian Dickson [mailto:[email-masked]]
Sent: Wednesday, June 27, 2012 5:06 PM
To: UTTARO, JAMES
Cc: Shane Amante; Senad .Palislamovic; [email-masked] List; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

On Wed, Jun 27, 2012 at 2:18 PM, UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> wrote:
All,

Two points here..

So IMO

Please indulge me, I am interested in understanding the relationship between the motivation/goal, the specific use cases presented, and the mechanisms themselves.

First, it seems that the "STALE" stuff is partly motivated by a perceived lack of notification in GR. Isn't that irrelevant to the draft at hand, and perhaps better addressed by updating GR?
[Jim U>] ? Not sure I know what you mean.. GR does not have the ability to inform topologies about paths learned over a session where GR is currently Active.. IMO this is a base requirement as the same path learned over a different egress PE should be preferred.. Whether this behavior "fits" with the GR spec I leave to the authors to comment on..

Also, in the use cases, there does not seem to be any real value to advertising this "STALE" state beyond the local systems. In the L2VPN in particular, there is no "beyond" at all. So, why bother with "STALE" at all? I don't see it being used anywhere in your draft, only that it is being set.
[Jim U>] In certain use cases dealing with static configuration this is certainly true, I would say exactly true when the update is specific action i.e FlowSpec, RT-C etc... VPLS uses BGP to set up PWs while the data plane is used  like ethernet to "learn" about MAC reachability. That being said there are a number of corner cases in VPLS i.e Multi_homing and some inter-AS scenarios where is may be prudent to either DO_NOT_PERSIST or to de-pref the path.. To be clear, the STALE and DO_NOT_PERSIST provide the operator with tools to realize their individual goals..

The main problem (that needs to be solved, i.e. the motivation for Persistence) seems to be that intra-AS PE routers, doing L2VPN, would like to maintain state in the case where one or more intermediate RRs "go away" for some period of time.
The scope of "state" information, is more closely aligned with the BGP peer state, rather than attributes of individual NLRIs of any given AFI/SAFI.
[Jim U>] Do not agree.. I think Persistence should be done at a path level so that we have control...The use case use mention is valid, there are others considered and yet to be considered.

Would it not be (a) simpler, (b) more scalable, and (c) more trivial to implement, by creating an Internet-Draft for a negotiated option, "Persistent", where, when two or more parties negotiate this option, and NLRIs heard over such sessions being "held" if/when a BGP session is terminated?
[Jim U>] Why? The STALE attr is standardized, everyone knows about it, operator takes action appropriate to their network.. What could be simpler??

This would not have any direct impact to path selection, for example, and would fairly trivially support negotiated (based on local configuration of) timers and "reaping" NLRIs upon timer expiry. Behavior per-AFI/SAFI should be able to be specified.
[Jim U>] We require the ability to de-pref.

Not needing to change the content or format of UPDATEs should be seen as a goal, rather than the other way round.
[Jim U>] ?

Let alone further overloading the COMMUNITY attribute, with mandated behavior based on something that is both optional _and_ transitive. (Yuck.)
[Jim U>] ?

In fact, would not this fit nicely as a sub-code to GR, with a negotiated status and timer between peers (or RR servers/clients)?
[Jim U>] GR does not meet the reqs.

Perhaps having the "negotiation" have multiple values, such as "never", "allow", "prefer", "always", so that in appropriately configured networks, on a per-peer (or peer-group) basis, it could be centrally controlled by the RR primarily.

One other question I have, is about Section 6.1.
I do not understand a couple of things about the diagram and text:
Is the diagram just messed up a bit, with RR2 meaning to talk to PE1 and PE2, rather than RR2 talking to  RR1 and CE1?
[Jim U>] The diagram got shifted to the left.. I will correct Thx..

Which BGP sessions are presumed to go away, i.e. where Persistence will make a difference (or fix a problem you're seeing)?
Is it the case that *both* PE1-RR1 *and* PE1-RR2 BGP sessions fail simultaneously?
[Jim U>] Not simultaneously.. For this use case the PE is peered to a cluster of RRs.. If there is a loss of connectivity from the PE to the Cluster then Persistence would kick in..

[Jim U>] Yes

Would the problem you are experiencing not be possible to fix with, say, PVRST over dual-PE1's facing CE1, and similarly dual-PE2's facing CE2, in a redundant configuration? That way, only PE1a (and not PE1b) would dump BGP peering, and things would fail-over to PE1b? (At which point it is a hardware cost and network design exercise, rather than a protocol thing.)
[Jim U>] I am not familiar with this.. That being said, I want one solution across a number of AFs i.e L3VPN, L2VPN, Flowspec, RT-C, 3107 etc....

You'd have two wires (PE1a-PE1b, and PE2a-PE2b) and two pseudo-wires (PE1a-PE2a, ,and PE1b-PE2b) forming a "square" topology on which some spanning-tree thing would need to turn into a loop-free topology, and it wouldn't matter which PE1 or PE2 was preferred by CE1 and CE2 respectively.

Sincerely,
Brian Dickson

---

From: Shane Amante <[email-masked]>
Date: Thu, 28 Jun 2012 20:14:51 -0600

Jim,

On Jun 28, 2012, at 5:58 PM, UTTARO, JAMES wrote:
> Jim U> I believe that different use cases require different behaviors. There is simply no way of getting around this.. As I stated in my ex.. Flowspec is a method of creating configuration via BGP.. This should persist as if it was programmed via XML, etc…  I always want robustness and scalability but certain AFs require more than this. I think the charter should to be amended to accomodate the different use cases for BGP and where they diverge in terms of behavior, customer expectations and solutions, or at minimum try to operate from a framework that acknowledges these use cases and their different requirements.

I'm not sure why you believe that the Internet should be treated as a "lesser citizen" in the context of requirements related to the robustness of BGP.  As a case in point, on the Internet, there exists a variety of OTT VoIP services, (e.g.: Vonage, Ooma, MagicJack, etc.), over which E911 services are provided.  That seems like a pretty critical application.

In addition, I'm not sure if you're aware (my apologies if you are), but recently the FCC here in the U.S., has started having discussions of setting a date for sunsetting the PSTN, which presumably means more customers would get pushed into using VoIP (of some form, whether as a service of their ISP or OTT) -or- mobile/cell phones:
(Please note the following is a link to a Word doc on the FCC's Web site, I could not find a link to a HTML version):
http://transition.fcc.gov/oet/tac/tacdocs/meeting92711/Sun-Setting_the_PSTN_Paper_V03.docx
The following is a link to a blog post summarizing the Word doc:
http://blog.connectedplanetonline.com/unfiltered/2011/07/07/fcc-considering-exploring-end-dates-for-the-pstn/

Interestingly enough, AT&T has apparently been requesting this of the FCC since, at least, back in 2009:
http://www.pcworld.com/businesscenter/article/185649/atandt_tells_fcc_its_time_to_cut_the_cord.html

Certainly there are other applications on the Internet that will likely emerge over time, which also will demand similar high availability/robustness of Internet service.  Thus, I say, let's take the opportunity now to find a common solution to a fundamental architectural problem of BGP, which will ultimately benefit all SP's and users of their services.

-shane

---

From: Brian Dickson <[email-masked]>
Date: Thu, 28 Jun 2012 22:58:31 -0400

Okay, thanks for answering my questions.

I'll try to simplify the discussion by splitting ongoing discussion into
separate threads.
I'll leave the subject unmodified, and hopefully modern mail readers can
cope with the references.

Brian

On Thu, Jun 28, 2012 at 9:36 PM, UTTARO, JAMES <[email-masked]> wrote:

>  Brian,****
>
> ** **
>
>                 Appreciate your comments. Comments In-Line..****
>
> ** **
>
> Thanks,****
>
> Jim Uttaro****
>
> ** **
>
> *
> *
>
> ** **
>
> be able to be specified.****
>
> *[Jim U>] We require the ability to de-pref.*****
>
> **
>

Can you elaborate?

Who is "we" in this context?

How strict is "require", and can you explain how that is a hard
requirement, maybe with examples of things breaking without de-pref?
Please, more use cases - the more the better. We need to fully understand
the problem space, with which you are intimately familiar.

If de-pref is not available, is "STALE" still useful?

If you are given a choice between "STALE but no de-pref" or "nothing",
would "STALE but no de-pref" be acceptable?

(Currently you have "nothing", and are asking for "STALE *and* de-pref". I
just want to be sure I understand the situation. I believe you could get
session-level STALE for the asking, with no opposition, as long as there is
no behavior modification or auto-depref mandated. I.e., if the behavior
was, "negotiate option PERSISTENCE. if SESSION.PERSISTENCE &&
BGP_SESSION_DROPPED, add community STALE, push the 'update-it' button which
then triggers out-bound route-maps et al", then I think you could get this
through WGLC in 30 days or less.)

> **
> goal, rather than the other way round.****
>
> *[Jim U>] ?*****
>
> **
>

A negotiated option, call it "PERSIST", would mean that your PE1 peerings
with RR1 and RR2, if negotiated, would keep the routes learned if the BGP
sessions dropped. Presume that in addition to "PERSIST" that the peers also
negotiate a "PERSIST_TIMER".

In order to do that, the CAPABILITY_NEGOTIATE codes need to be defined, but
beyond the OPEN, there is no change to the on-the-wire protocol or to the
path-selection process. The only additional code change needed would be the
implicit withdrawal when a BGP peer drops.

Contrast that, from the perspective of multiple implementers, of all the
mechanics involved in checking each UPDATE for the presence of STALE or
DO_NOT_PERSIST communities, and hard-coding modifications to behavior that
already involves so many moving parts.
(Believe me - I've done severe hacking on, e.g. quagga, and while it is
solid code, the complexity in it, which is informed directly by the
plethora of BGP RFCs, makes it a huge challenge to try to do "big" changes
to it on a whole cloth basis. Getting those right and debugging them are
highly non-trivial. Now hand that to vendors who have trouble managing
their own hardware drivers and code trains, with massive teams of
developers. This needs to interoperate, right? And be stable code serving a
lot of customers?)

Minimizing the number of places that already-complex logic needs to be
tweaked, and then looking at corner cases where those might or might not
overlap (in the code, not in the UPDATE data, that is)... If the benefits
(of your proposal) are only seen in corner cases, and if the benefits can
maybe be achieved via other means (voluntary COMMUNITY processing via
route-maps, plus maybe the setting of a STALE well-known community if/when
a session drops), then would you be willing to consider the simpler
solution?

(It might help to try to take what I've suggested and draw it out on a
white board. See if you can make sense of it and make it work, even if you
have to do post-processing stuff in route-maps. The question is, does it
provide sufficient mechanisms to achieve the design goals?)

Sincerely,
Brian Dickson

**

**

** **

---

From: Brian Dickson <[email-masked]>
Date: Thu, 28 Jun 2012 23:30:34 -0400

Second sub-thread...

Thanks for discussing these issues/questions.

Brian

On Thu, Jun 28, 2012 at 9:36 PM, UTTARO, JAMES <[email-masked]> wrote:

> make a difference (or fix a problem you're seeing)?****
>
> simultaneously?****
> *[Jim U>] Not simultaneously.. For this use case the PE is peered to a
> cluster of RRs.. If there is a loss of connectivity from the PE to the
> Cluster then Persistence would kick in..*
>

Does this mean that the cluster of RRs are in one place, where the
reachability of the RRs is in a "shared fate" scenario? (It's less severe
than "single point of failure", but less diverse/survivable than "fully
dispersed, topologically/geographically diverse RRs".)

Loss of connectivity to the Cluster => simultaneous BGP session failures.

The consequence is the same, regardless of cause. Simultaneous failures of
BGP sessions.

If the network design (single cluster of RRs) is at fault, while yes,
Persistence would help, it is definitely a corner case. If you don't have
an upper bound on network issues, how can you set (apriori) the values on
your timers for persistence? Or would they need to be settable unilaterally
on the PE?

Is there a fundamental property of the Cluster, that means you can't have
two Clusters to avoid the issue?

And, to be clear, the problem is not on the PE end per se, or simultaneous
bugs hitting RR1 and RR2, it is a network isolation thing between PE1 and
RR1/RR2, that is the frequent and problematic root issue for the problems
you are having in your L2VPNs?

Thanks,
Brian

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 29 Jun 2012 02:49:27 -0400

On Thursday, June 28, 2012 12:31 PM, John G.Scudder <mailto:[email-masked]> wrote:

> If by "will forward packets" you mean "... to their intended
> destination" this is not generally correct. Consider the use of
> more-specifics for prefix portability from one provider to another.
> The provider that supplied the prefix continues to own and advertise
> the supernet, but no longer provides service to their dear departed
> customer.     

Thanks John. Makes sense.

Thus, we can not depref these SUSPECT paths against
other paths that may exist (the less specific ones).

-- 
Jakob Heitz.

---

From: <[email-masked]>
Date: Fri, 29 Jun 2012 09:05:40 +0000

Brian,

Thanks for your comments.
Comments In-Line [BD]

Regards,
Bruno

From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Dickson
Sent: Friday, June 29, 2012 4:59 AM
To: UTTARO, JAMES
Cc: Shane Amante; Robert Raszuk; [email-masked] List
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

I'll try to simplify the discussion by splitting ongoing discussion into separate threads.
I'll leave the subject unmodified, and hopefully modern mail readers can cope with the references.

Brian
On Thu, Jun 28, 2012 at 9:36 PM, UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> wrote:
Brian,

Thanks,
Jim Uttaro

Can you elaborate?

How strict is "require", and can you explain how that is a hard requirement, maybe with examples of things breaking without de-pref?
Please, more use cases - the more the better. We need to fully understand the problem space, with which you are intimately familiar.

[BD] IMHO Yes.

If you are given a choice between "STALE but no de-pref" or "nothing", would "STALE but no de-pref" be acceptable?
[BD] IMHO Yes.
Because in the end, the remaining part can probably be done by the SP itself using a routemap (e.g. STALE & ! DO_NOT_PERSIST --> SET Local Pref= 0; SET NO_EXPORT)

(Currently you have "nothing", and are asking for "STALE *and* de-pref". I just want to be sure I understand the situation. I believe you could get session-level STALE for the asking, with no opposition, as long as there is no behavior modification or auto-depref mandated.
[BD] not sure what you mean by "no behavior modification". There is a significant change: keep the routes while the BGP session is dead & set STALE community
I.e., if the behavior was, "negotiate option PERSISTENCE. if SESSION.PERSISTENCE && BGP_SESSION_DROPPED, add community STALE, push the 'update-it' button which then triggers out-bound route-maps et al",
[BD] within iBGP, IMHO there may not even a need for a negotiation

then I think you could get this through WGLC in 30 days or less.)
 [BD] Looks like not :)
But indeed there are a few questions which need to be discussed:
- how much is this applicable to all services (or AFI/SAFI) or only specific ones
IMHO this relates to how much control/dynamicity we have on the routes. Which could probably translate in determining a duration which is considered relatively safe. Idem GR. Idem this could/should be left to the SP. But indeed, if for a service the "right" value is 5 minutes then the benefit is limited.
- how much frequent/rare. I.e. does this justify to work on this / change the implementation.
IMHO Quite personal opinion. No comment.
That's about probability. I don't know the future. In some way, a bit similar to insurance policy.
- this may be risky. There must be a better solution.
             Indeed. But this is also a very generic safety net which may be expected to catch lots of issues, including the next -not expected- ones. You're very welcomed to propose a better (complementary or competing) solution. IMHO persistence should not replace more specific solutions focusing on a specific issue.
Not needing to change the content or format of UPDATEs should be seen as a goal, rather than the other way round.
[Jim U>] ?

A negotiated option, call it "PERSIST", would mean that your PE1 peerings with RR1 and RR2, if negotiated, would keep the routes learned if the BGP sessions dropped. Presume that in addition to "PERSIST" that the peers also negotiate a "PERSIST_TIMER".

In order to do that, the CAPABILITY_NEGOTIATE codes need to be defined, but beyond the OPEN, there is no change to the on-the-wire protocol or to the path-selection process. The only additional code change needed would be the implicit withdrawal when a BGP peer drops.

Contrast that, from the perspective of multiple implementers, of all the mechanics involved in checking each UPDATE for the presence of STALE or DO_NOT_PERSIST communities, and hard-coding modifications to behavior that already involves so many moving parts.
(Believe me - I've done severe hacking on, e.g. quagga, and while it is solid code, the complexity in it, which is informed directly by the plethora of BGP RFCs, makes it a huge challenge to try to do "big" changes to it on a whole cloth basis. Getting those right and debugging them are highly non-trivial. Now hand that to vendors who have trouble managing their own hardware drivers and code trains, with massive teams of developers. This needs to interoperate, right? And be stable code serving a lot of customers?)

Minimizing the number of places that already-complex logic needs to be tweaked, and then looking at corner cases where those might or might not overlap (in the code, not in the UPDATE data, that is)... If the benefits (of your proposal) are only seen in corner cases, and if the benefits can maybe be achieved via other means (voluntary COMMUNITY processing via route-maps, plus maybe the setting of a STALE well-known community if/when a session drops), then would you be willing to consider the simpler solution?

(It might help to try to take what I've suggested and draw it out on a white board. See if you can make sense of it and make it work, even if you have to do post-processing stuff in route-maps. The question is, does it provide sufficient mechanisms to achieve the design goals?)

Sincerely,
Brian Dickson

---

From: "John G. Scudder" <[email-masked]>
Date: Fri, 29 Jun 2012 10:35:45 -0400

Yes. I was specifically referring to "keep the session up even if some errors which would trigger it down are received", which I took to mean the error handling draft and related work. 

Graceful Restart can indeed address transport-and-below issues, but then we need to review the discussion of the specific differences between persistence and GR, notably the decision to withhold any signaling about the outage (GR) vs. depreffing the stale routes (persistence) and the applicability of the respective strategies given different time windows. I don't have anything new to say about that.

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 29 Jun 2012 17:43:34 +0000

Shane,

                I agree,  there will be applications that use the internet service requiring persistence and I believe the technical specification BGP Persistence should be used.. That being said there are unique challenges that come to mind:

a) Advertisements.. When Persistence is active, a speaker cannot inform the topology of changes to the state.. For static services i.e L2VPN, or L3VPN which is considered less dynamic than the internet this inability is a good tradeoff as we know the state does not change often and therefore the incorrectness will be minimal. Regardless the customer would have the ability to DO_NOT _PERSIST state..

b) IMO Persistence is most applicable where forwarding and control plane are decoupled.. Not an internet person but in the case where a speaker is a NH for paths and advertiser for those paths is it an appropriate topology for Persistence??  I may be wrong, to be honest I have not considered the internet use case enough to be sure..

c) From the responses on the forum it is clear that there are very strong feelings in re changing the behavior in the internet.. Too be honest I felt that the discussions would become religious if the internet was included . I really want to advance this draft for a number of applications where I do not have alternative paths via other peering points, and the state is quite static.

I in no way meant to infer that the internet was a "lesser citizen" I was only attempting to convey that there is a requirement for Persistence for other services which has never been  a requirement for the Internet use case. I believe that the services over the internet ( As you point out below ) will require Persistence.

Jim Uttaro

From: Shane Amante [mailto:[email-masked]]
Sent: Thursday, June 28, 2012 10:15 PM
To: UTTARO, JAMES
Cc: Senad .Palislamovic; [email-masked] List; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Jim,

---snip---

-shane

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 29 Jun 2012 17:49:38 +0000

John,

	One additional comment here in re GR and Persistence. We need to include the discussion in re the failure modes and how GR and Persistence react.. As an ex.. Subsequent Restart, Timer Expiry, Mal-Formed update failures force GR to flush the state as it is assumed to be a failure mode that should not be survivable.. Persistence does not flush the state when the session "fails" due to these conditions...

Jim Uttaro

-----Original Message-----
From: John G. Scudder [mailto:[email-masked]] 
Sent: Friday, June 29, 2012 10:36 AM
To: [email-masked]
Cc: Senad .Palislamovic; Shane Amante; [email-masked]; UTTARO, JAMES; Susan Hares; [email-masked]
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

--John

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 29 Jun 2012 17:56:08 +0000

Brian,

                Comments In-Line...

Jim Uttaro

From: Brian Dickson [mailto:[email-masked]]
Sent: Thursday, June 28, 2012 11:31 PM
To: UTTARO, JAMES
Cc: Shane Amante; Senad .Palislamovic; [email-masked] List; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Second sub-thread...

Brian

Thanks,
Jim Uttaro

Does this mean that the cluster of RRs are in one place, where the reachability of the RRs is in a "shared fate" scenario? (It's less severe than "single point of failure", but less diverse/survivable than "fully dispersed, topologically/geographically diverse RRs".)
[Jim U>] No.. The Cluster has RRs its RRs in diverse geography and topology..

[Jim U>] True

The consequence is the same, regardless of cause. Simultaneous failures of BGP sessions.
[Jim U>] Yes

If the network design (single cluster of RRs) is at fault, while yes, Persistence would help, it is definitely a corner case. If you don't have an upper bound on network issues, how can you set (apriori) the values on your timers for persistence? Or would they need to be settable unilaterally on the PE?
[Jim U>] Interesting question.. I believe for configuration type services the upper bound should be set quite high.. For L3VPN probably lower as there is a greater risk of incorrectness..

Is there a fundamental property of the Cluster, that means you can't have two Clusters to avoid the issue?
[Jim U>] Although at first blush it would seem that adding addl redundant clusters would solve the problem it can only mitigate the effect as the damage can be localized.. At some point we get to a full mesh which is unsustainable...

And, to be clear, the problem is not on the PE end per se, or simultaneous bugs hitting RR1 and RR2, it is a network isolation thing between PE1 and RR1/RR2, that is the frequent and problematic root issue for the problems you are having in your L2VPNs?
[Jim U>] Not limited to L2VPNs.. Could be bugs, could be topological, could be scale, etc....We have seen different triggers.

Thanks,
Brian

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 29 Jun 2012 18:02:33 +0000

Brian,

                Comments In-Line..

Jim Uttaro

From: Brian Dickson [mailto:[email-masked]]
Sent: Thursday, June 28, 2012 10:59 PM
To: UTTARO, JAMES
Cc: Shane Amante; Senad .Palislamovic; [email-masked] List; Robert Raszuk
Subject: Re: [Idr] draft-uttaro-idr-bgp-persistence-01 as IDR WG document - 3 more weeks

Brian

Thanks,
Jim Uttaro

Can you elaborate?
[Jim U>] If there is another path in the topology then that should be used..

Who is "we" in this context?
[Jim U>]I guess I will only speak for myself here..

How strict is "require", and can you explain how that is a hard requirement, maybe with examples of things breaking without de-pref?
Please, more use cases - the more the better. We need to fully understand the problem space, with which you are intimately familiar.
[Jim U>] I guess it is important to note that the draft provides two mechanisms, the first is de-pref the second is DO_NOT _PERSIST.. These tools allow the SP to have different behaviors.. It may also be appropriate to simply Persist the state without de-preffing although that does not seem right if there is a path over a session which is not experiencing issues..

If de-pref is not available, is "STALE" still useful?
[Jim U>] Yes, this would be the case where that is the only path available or better said the content of the update is not about paths but about configuration..Too be clear configuration BGP needs further examination..

If you are given a choice between "STALE but no de-pref" or "nothing", would "STALE but no de-pref" be acceptable?
[Jim U>] Not really sure why I have to have a choice.. We are using LP to de-pref..

(Currently you have "nothing", and are asking for "STALE *and* de-pref". I just want to be sure I understand the situation. I believe you could get session-level STALE for the asking, with no opposition, as long as there is no behavior modification or auto-depref mandated. I.e., if the behavior was, "negotiate option PERSISTENCE. if SESSION.PERSISTENCE && BGP_SESSION_DROPPED, add community STALE, push the 'update-it' button which then triggers out-bound route-maps et al", then I think you could get this through WGLC in 30 days or less.)
[Jim U>] That is quite optimistic ;)

Sincerely,
Brian Dickson