Subject: [GROW] Repeated Errors in BGP - draft-ietf-grow-ops-reqs-for-bgp-error-handling

From: Rob Shakir <[email-masked]>
Date: Fri, 13 Apr 2012 19:27:36 +0100

Hi All,

I'd like to ask the WG their collective opinion on a couple of matters in this draft, which come from some discussions at IETF83 (in particular with John Scudder and Adam Simpson) about how the requirements are currently written regarding repeated errors.

The wording right now in the draft implies that at some point, the speaker that detects the failure could push the session into a "hold up" state ignoring any further UPDATE errors (i.e. at some point the 'critical' error in the UPDATE is just ignored) and the prefixes that are are there are maintained.

   In the case of implementation errors, it is
   possible that the BGP session in question may enter a continuous loop
   of being reset, with a partial RIB being held by one or more of the
   BGP speakers due to an non-deterministic order of UPDATE propagation.
   It is therefore a requirement that within this reduced-impact
   procedure any subsequent UPDATE messages that would result in further
   session resets are ignored.  Whilst this results in a condition where
   an undetermined amount of the RIB is inconsistent, partial
   reachability is maintained.  In this case, the operational toolsets
   discussed in Section 6 is likely to provide mechanisms by which this
   condition can be brought to the attention of the relevant operators.
   This requirement to accept a partial RIB, which results in potential
   invalid traffic forwarding is a direct result of the deployments of
   BGP-4, as described in Section 1.1.

I think that actually, the discussions that I've had with various WG participants relating to this imply that at some point instead of ignoring the UPDATE, the session should be considered no longer viable, and the session ceased.

I think that the latter view is relevant only to Critical errors (i.e. those whereby the NLRI cannot be identified) that cause session reset - and that the implication should be at that at some point there is a requirement /not/ to continually keep applying the mechanisms that are specified in Section 5 of the draft. This is the behaviour that is implied in Section 7.1 of the draft.

The original wording of the draft did not draw a distinction between the behaviour expected for Semantic and Critical errors - which I think meant that the above recommendation was more relevant (one would not want to teardown based on errors that could be handled with treat-as-withdraw). Given that repeated Critical errors require repeated session-level error handling, this would seem to be indicative of a relatively long-lived failure (which is unlikely to recover) and hence at some point, continually performing hitless restarts becomes a futile exercise.

So, as such, given that there is an internal lack of consistency between these two sections - the suggestion I am proposing is to restructure Section 7 of the draft, such that it recommends:

- For Semantic errors, a staged recovery approach should be taken (should any automatic recovery be attempted) such that a specific recovery mechanism is implemented, in preference to re-requesting the entire RIB (so as to reduce the number of messages that are received).
- Where such Semantic errors are repeated, at some point (defined per-recovery mechanism memo or on a per-implementation basis based on the resource consumption of the recovery mechanism), no further automatic refresh action should be taken as a result of further Semantic errors. Such a condition should be flagged to an operator to flag this has taken place.
- For repeated Critical errors, an implementation maintains some "session badness" flag, which can be used to work out that the "hitless session restart" mechanisms (§5) should no longer be applied. This condition should be flagged to an operator.
- There is no requirement within the "hitless session restart" mechanism to have any form of delay in terms of re-opening the session, since this must occur inside the time bound for the session restart described in Section 5.
- (As per John's suggestion) It is recommended that there is an exponential back-off for session re-establishment following the decision to stop using hitless restart procedures for a session (as per the existing definition for peer oscillation in RFC4271).

Is the WG in agreement with this recommendation, and restructuring of this section?

Many thanks,
r.

---

From: Tony Li <[email-masked]>
Date: Fri, 13 Apr 2012 13:24:42 -0700

On Apr 13, 2012, at 11:27 AM, Rob Shakir wrote:

>   session resets are ignored. 

Hi,

This may be a nit, but I think it's important to recall that UPDATE messages include withdrawn routes.  These MUST NOT be ignored by the receiver.  Doing so will simply result in forwarding loops or black holes.

Perhaps this should at least be wordsmithed into something like 'ignore any reachability information in an UPDATE message, while processing the withdrawn routes in that same UPDATE message'.

Independently of that, I think that trying to maintain a session in the face of multiple errors is a clear waste of time and effort on all parties.  At some point, there is more effort and complexity spent on error recovery than on correct transmission, and that's just backwards.  I support the suggestion of binary exponential back off on session restarts.

Regards,
Tony

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 15 Apr 2012 17:45:49 +0100

On 13/04/2012 21:24, Tony Li wrote:
> This may be a nit, but I think it's important to recall that UPDATE
> messages include withdrawn routes.  These MUST NOT be ignored by the
> receiver.  Doing so will simply result in forwarding loops or black
> holes.

I'm trying to figure out which is worse: accepting a semantically known
corrupt UPDATE withdraw message or ignoring it.  Also, I suspect that this
is getting into non-deterministic territory.  If you're seeing a continuous
stream of corrupt updates for different prefixes from a peer, that probably
indicates a more serious problem, for which session teardown might be a
more appropriate fix than ignoring broken nlri messages.

> Perhaps this should at least be wordsmithed into something like 'ignore
> any reachability information in an UPDATE message, while processing the
> withdrawn routes in that same UPDATE message'.

> Independently of that, I think that trying to maintain a session in the
> face of multiple errors is a clear waste of time and effort on all
> parties.

this is an important point: at what stage do you give up on a session?  The
draft has some discussion on this, but it might be useful to have a clearer
statement on what to do when you see a continuous stream of garbage.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 16 Apr 2012 05:06:45 +0200

> Independently of that, I think that trying to maintain a session in
> the face of multiple errors is a clear waste of time and effort on
> all parties.  At some point, there is more effort and complexity
> spent on error recovery than on correct transmission, and that's just
> backwards.  I support the suggestion of binary exponential back off
> on session restarts.
>
> Regards, Tony

I would highly agree that the notion to keep the session at all cost is 
just a wrong thing to do.

Such notion comes often from the fact that operator did not provided 
path redundancy or are trigger by seemingly correct observation that 
single malformed update should not impact other correct updates received 
over the same session.

However neither the error handling IDR spec nor it's early 
implementations provide a way to still allow the reset to happen if X % 
of UPDATE messages in the window of time T sec are malformed.

Perhaps Rob's document as set of requirements could add such requirement 
to the spec ?

Regards,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 15 Apr 2012 23:22:06 -0400

IMO
1. Error handling must be simple, lest we have to handle errors
of the error handling, ad infinitum...

2. Error handling should be restricted to error isolation
and reporting. It should refrain from attempting error recovery.

If an error can be isolated to a set of NLRI, withdraw them.
If an error can only be isolated to a session, reset it.

Tracking and analysing multiple errors to decide on a reset
violates point 1.

On Sunday, April 15, 2012 8:07 PM, Robert Raszuk <> wrote:

> I would highly agree that the notion to keep the session at all cost
> is just a wrong thing to do.
> 
> single malformed update should not impact other correct updates
> received over the same session.
> 
> implementations provide a way to still allow the reset to happen if X
> % of UPDATE messages in the window of time T sec are malformed.
> 
> Perhaps Rob's document as set of requirements could add such
> requirement to the spec ?
> 
> Regards,
> R.

-- 
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 16 Apr 2012 05:33:40 +0200

Hi Jakob,

Just to make sure you got my point ...

I am not suggesting adding ton of logic and algorithmic operations.

All I am suggesting is simple counter (time drained) which would count 
how many times in a given period we are performing the treat-as-withdraw 
action.

Algorithm:
----------

If such counter get's exceeded session get's reset.

There is tracking (counter++ operation) and one additional conditional 
if statement. There is no analysis what so ever.

Best,
R.

> On Sunday, April 15, 2012 8:07 PM, Robert Raszuk<>  wrote:
>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 15 Apr 2012 23:40:10 -0400

Your algorithm has an error.
You forgot to reset the counter after resetting the session.
How will you handle that error?

Software is not that easy.
Get my point?

On Sunday, April 15, 2012 8:34 PM, Robert Raszuk <mailto:[email-masked]> wrote:

>> Tracking and analysing multiple errors to decide on a reset violates
>> point 1. 
>> 
>>>> spent on error recovery than on correct transmission, and that's
>>>> just backwards.  I support the suggestion of binary exponential
>>>> back off on session restarts. 
>>>> 
>>>> Regards, Tony
>>> 
>>> I would highly agree that the notion to keep the session at all
>>> cost is just a wrong thing to do. 
>>> 
>>> implementations provide a way to still allow the reset to happen if
>>> X % of UPDATE messages in the window of time T sec are malformed.
>>> 

-- 
Jakob Heitz.

---

From: Tony Li <[email-masked]>
Date: Sun, 15 Apr 2012 20:42:46 -0700

+1

Tony

On Apr 15, 2012, at 8:22 PM, Jakob Heitz wrote:

> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 16 Apr 2012 05:55:35 +0200

> How will you handle that error?

No it does not. This counter is part of dozen of other per session 
counters and variables which get reset and cleared when session goes 
down. No need to for any specific handling of just this counter.

R.

> On Sunday, April 15, 2012 8:34 PM, Robert Raszuk<mailto:[email-masked]>  wrote:
>
>> Hi Jakob,
>>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 15 Apr 2012 23:59:49 -0400

> down. No need to for any specific handling of just this counter.

We all say "Oh, I meant to do that" when we find a bug.

>>> All I am suggesting is simple counter (time drained) which would
>>> count how many times in a given period we are performing the
>>> treat-as-withdraw action.
>>> 
>>> Algorithm:
>>> ----------
>>> 
>>> There is tracking (counter++ operation) and one additional
>>> conditional if statement. There is no analysis what so ever.
>>> 
>>> Best,
>>> R.
>>> 
>>> 
>>>> IMO
>>>>>> Independently of that, I think that trying to maintain a session
>>>>>> in the face of multiple errors is a clear waste of time and
>>>>>> effort on all parties.  At some point, there is more effort and
>>>>>> complexity spent on error recovery than on correct transmission,
>>>>>> and that's just backwards.  I support the suggestion of binary
>>>>>> exponential back off on session restarts. 
>>>>>> 
>>>>>> Regards, Tony
>>>>> 
>>>>> Such notion comes often from the fact that operator did not
>>>>> provided path redundancy or are trigger by seemingly correct
>>>>> observation that single malformed update should not impact other
>>>>> correct updates received over the same session.
>>>>> 
>>>>> implementations provide a way to still allow the reset to happen
>>>>> if X % of UPDATE messages in the window of time T sec are
>>>>> malformed. 
>>>>> 

-- 
Jakob Heitz.

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Mon, 16 Apr 2012 09:16:34 +0200

Robert,

> -----Original Message-----
> Algorithm:
> ----------
> 
> There is tracking (counter++ operation) and one additional conditional if
> statement. There is no analysis what so ever.
> 

Let's consider following scenario: a router in no-longer-kingdom far-away
sends malformed update that somehow manages to slip through multiple
intermediate routers and comes to me via a peer that sends me few dozen
thousands of other prefixes (everyone has seen this in the wild not too long
ago). I increase "error tracking counter", the far-away router for some
reason sends update again, I increase counter again, at some point I reach
threshold. Clearly, resetting my session with a peer that sends me large
number of "good" prefixes isn't to my advantage, especially that it won't
repair distant router. My personal preference would be to ignore such errors
until kingdom come.

On the other hand, if a session is reset (due to errors per other rules)
there's chance that problems will occur again. Rather than bouncing session
for extended period, I'd prefer such troublesome peers to stay down. We
spoke about this behaviour few times, but I'm not sure if we have reached
consensus.

/iLya

---

From: Rob Shakir <[email-masked]>
Date: Mon, 16 Apr 2012 09:53:18 +0100

Hi,

Thanks for the feedback.

> Perhaps this should at least be wordsmithed into something like 'ignore any reachability information in an UPDATE message, while processing the withdrawn routes in that same UPDATE message'.

Agreed - I think that this is the only place in the draft where there is any reference to ignoring a message, but let me re-review this and take your suggestion into account.

> Independently of that, I think that trying to maintain a session in the face of multiple errors is a clear waste of time and effort on all parties.  At some point, there is more effort and complexi…

This was the original intent of the section, to ensure that at some point there is a cap on the amount of resource that goes into handling these errors. 

The problem with defining an arbitrary cap of "X errors in N time" (as Robert suggested) is that this trips up on some particular scenarios:

- A remote AS in the Internet DFZ re-announces prefixes due to upgrading border routers, with some changed attribute - it's likely that all their prefixes are learnt within a short time period, and if all of them include some semantic error, then transitioning to a down state will impact the rest of the prefixes in the DFZ unnecessarily.

- If a PE in an L3VPN environment experiences some event (be it configuration change, software upgrade, etc) and as a result propagates prefixes which are made invalid by an element of the RR infrastructure, then a all prefixes will be readvertised within a relatively short time period. Again if this results in transitioning the session to a down state on the receiving PE (receiving from the RRs), then this impacts all prefixes from other PEs unnecessarily.

In both cases these are lower-impact to the receiving PE if they are semantic errors, since the action required is likely to be the generation of an UPDATE withdrawing these prefixes. Where automatic recovery is put in place by an implementor, really the requirement is to cap how much of the receiving speaker's resource is used - hence the suggestion in the original mail in this thread.

As I see it, for critical errors, where the hitless restart is the way of handling the error, then this is somewhat different due to the somewhat heavier impact of the receiving PE, plus the fact that the error is much more likely to be localised to the directly peered-with speaker, and hence the cap of resource by shutting the session down at some point, and then exponentially backing off the re-opening process.

Please let me know if this doesn't sound reasonable, or whether there's any assumption I made here that's invalid.

Cheers,
r.

---

From: Rob Shakir <[email-masked]>
Date: Mon, 16 Apr 2012 10:05:33 +0100

Hi Jakob,

I agree with this, but would like to clarify something here.

I think that there is a requirement to ensure that we do not overwhelm the resources of a particular PE by constant session restarts. If one were to implement NOTIFICATION-based G-R, when do you stop trying to G-R and instead assume that there is something that is not going to go away on the other side?

If the answer is "never", then I guess the requirement outlined in the section I'm discussing is moot. However, from my POV, there's definitely a need to say that at some point my box should stop trying to expend effort fixing things (hence the original wording with 'hold up', and now the discussion of 'shutdown and back off').

Cheers
r.

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 16 Apr 2012 07:50:49 -0400

This should not overwhelm a router.
However, a more serious consequence is a lot of flapping routes.
Serious enough to consider. We could dust off some dampening code for it.

Note the previous discussion was about repeated "treat as withdraw"
errors. I support the use of an operational message for these errors,
but nothing for repetitions of them. Repeated operational messages
may get annoying, but they are not going to disrupt service.

--
Jakob Heitz.

---

From: Jared Mauch <[email-masked]>
Date: Mon, 16 Apr 2012 09:35:31 -0400

> may get annoying, but they are not going to disrupt service.

The problem we have today is two fold:

1) Vendors have done a bad job of reporting the routes related to a BGP session reset
  a - multiple vendor environments can have somewhat catastrophic failure modes

2) Stability of the internet (as defined by "keeping the BGP session up for the non-broken routes") is important; overall reachability is as well.  (to Tony Li's comment about how an update has both feasible and infeasible nlri in it).

A broken implementation is just that, broken.  About half the problems in the past 24 months are due to improper parsing of a well formed update.  With the balance the improper propagation of a invalid update.  (I'm willing to be corrected on the exact ratio, but I believe these are the two cases seen unless my memory fails me).

I'm hard pressed to imagine a case where the code doesn't quickly get unwieldily from this situation.

Part of the requirements of solving this problem will be:

a) Sending an UPDATE message with only new/updated NLRI and zero items in the withdrawl.
b) Sending a KEEPALIVE (or other well known message that will be parsed properly).
c) Sending an UPDATE message with no new NLRI and only those to be withdrawn.

Perhaps this mode is only triggered after a previous session drop within 3600 seconds.   This means you need to track that state, track a timer, and utilize this new (capability?) mode.

We also have a more systematic issue here.  Those participating in the global BGP ecosystem need to play an active role in the maintenance of these devices.  Failure to do so can and will harm the rest of the ecosystem.  I'm not sure the best way to enforce this operationally but disabling sessions must be an operational response.  The reason for the reset is to *cause pain* for the poorly behaving devices in question and draw attention to them.  The same reason a developer would call abort().

The case of dropping the session is the developer equivalent of abort(), meant to alert the users to the problem.  Since not everyone running BGP is a protocol expert these days, I almost feel we need strict guidance and requirements around the logging we need to report these problems to the vendors.

- Jared

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 16 Apr 2012 21:43:35 +0200

Ilya,

I meant to say that you would increment the counter only for unique 
updates. So duplicate updates with the same NLRIs would not be counted N 
times.

 > On the other hand, if a session is reset (due to errors per other
 > rules) there's chance that problems will occur again. Rather than
 > bouncing session for extended period, I'd prefer such troublesome
 > peers to stay down. We spoke about this behaviour few times, but I'm
 > not sure if we have reached consensus.

I agree. We could have max-prefix like configurable restart as an option.

Rgs,
R.

---

From: Tony Li <[email-masked]>
Date: Mon, 16 Apr 2012 12:49:51 -0700

So you're going to _remember what error occurred_????

Complexity++

No thanks,
Tony

On Apr 16, 2012, at 12:43 PM, Robert Raszuk wrote:

> Ilya,
> 
> I meant to say that you would increment the counter only for unique updates. So duplicate updates with the same NLRIs would not be counted N times.
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 17 Apr 2012 00:01:19 +0200

Nope.

This is about recognizing duplicate update which good implementation 
should be able to do regardless.

Nothing to do with error itself.

Regards,
R.

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 16 Apr 2012 20:24:29 -0400

On Mon, Apr 16, 2012 at 6:01 PM, Robert Raszuk <[email-masked]> wrote:
> Nope.
>
> This is about recognizing duplicate update which good implementation should
> be able to do regardless.

I think 'good implementations' don't then exist, since ... as
shane/danny have presented a few times, a large percentage of update
traffic seen is repeats from the same speaker.

I'm not sure that being able to track repeats is really necessary
though, if one speaker is +10% bad (pick a percent) over the last 5
mins (or 500 update messages, perhaps this is a chance for a knob for
either/both/mix) shutdown the session and keep it down (or bounce and
re-establish and after X bounces in Y time, shutdown)

If the concern is, this peer keeps sending me an update that is
unparsable, ignoring it (and logging a note/alert/etc) seems ok. If
the problem is that the neighbor did something quite wrong on the
session as a whole more immediate action should be taken.

There's some sympathy to be had for situations like ATTR-128 though,
why bounce 50% of the speakers on the internet repeatedly if some
downstream of a downstream screws up?

-chris

>>>>> if

---

From: Tony Li <[email-masked]>
Date: Mon, 16 Apr 2012 17:37:02 -0700

> session as a whole more immediate action should be taken.

Sending something unparsable *is* quite wrong and just ignoring it is going to lead to errors down the road.  See previous comment about withdrawn routes in updates.

Tony

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 17 Apr 2012 21:35:22 +0200

> re-establish and after X bounces in Y time, shutdown)

I'd propose a variation of above:

if number of prefix in "bad update" is higher than configured X% of all prefixes (alternatively - 'of best path') learned from this peer (regardless of time) then reset session. If X configured to be 100, then effectively it means "never drop this session" (unless really all that this peer sends is bad).

/iLya

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 17 Apr 2012 23:00:52 +0200

Hi Ilya,

I very much agree with your adjustments. My goal was to just provide an 
example of what one could do.

Then to play devil's advocate one could observe that we should perhaps 
not bind the updates to sessions at all. In fact persistence draft is 
just about cutting that binding.

I guess the fact that we use TCP for BGP is sort of legacy and perhaps 
the more Shakespeare we put in BGP (on 179 or different instance) we 
should revisit how we transport the UPDATE messages betwen BGP peers.

Best,
R.

---

From: Tony Li <[email-masked]>
Date: Tue, 17 Apr 2012 14:26:17 -0700

On Apr 17, 2012, at 2:00 PM, Robert Raszuk wrote:

> I guess the fact that we use TCP for BGP is sort of legacy and perhaps the more Shakespeare we put in BGP (on 179 or different instance) we should revisit how we transport the UPDATE messages betwen BGP peers.

Before you go there, you need to ask yourself if you want to continue to have a protocol with incremental updates.  TCP is there precisely to support that.

If you want to go back to periodic updates, then getting rid of TCP seems much more credible.

But then the overhead become untenable.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 17 Apr 2012 23:47:51 +0200

> But then the overhead become untenable.

So why not event driven rsync like mechanism between BGP_Adj_Ribs for
incremental updates ? *well for VPNs RTC would need to be mandatory*

Note that RPKI is rsync based which some suggest for BGP to relay on 
anyway ;)

r.

>> I guess the fact that we use TCP for BGP is sort of legacy and
>> perhaps the more Shakespeare we put in BGP (on 179 or different
>> instance) we should revisit how we transport the UPDATE messages
>> betwen BGP peers.
>
> Before you go there, you need to ask yourself if you want to continue
> to have a protocol with incremental updates.  TCP is there precisely
> to support that.
>

---

From: Tony Li <[email-masked]>
Date: Tue, 17 Apr 2012 15:12:06 -0700

> incremental updates ? *well for VPNs RTC would need to be mandatory*

That doesn't scale particularly well.  The time needed to 'sync' is still linear with respect to the number of prefixes, and you end up sending a digest of your database. When the items involved gigabit files, this is an efficiency win.  When the routes are in the hundreds of bytes, you're saving maybe 2 orders of magnitude over the entire database.  

Sure, that's better, but that's a fixed, one time multiplier on scale.  Sooner or later, you'll exhaust that.

> Note that RPKI is rsync based which some suggest for BGP to relay on anyway ;)

My recollection is that this was for static key information.  Only slightly different.  ;-)

Tony

---

From: heasley <[email-masked]>
Date: Tue, 17 Apr 2012 17:47:44 -0700

> Sending something unparsable *is* quite wrong and just ignoring it is going to lead to errors down the road.  See previous comment about withdrawn routes in updates.

We agree with Tony.  Furthermore, if an update has a parsing or semantic
error, what degree of confidence does one have that any portion that appears
to be parsable is in fact correct?

Obviously, there is merit in the problem statements, but the assumption that
one could trust any part of a message exhibiting any errors seems like a
rather poor choice.  Without fundamental changes in how updates are handled,
route-refresh seems like the only reliable way short of a reset to ensure
consistency in recovering from a malformed update.

Seems that effort is best directed at improving GR, ISSU, logging (syslog,
more specific notifications & similar) and monitoring capabilities such as
BMP that allow evaluation and tracking on a management host and other tools
that aid in event diagnosis and isolation by operations folk.  Developing
negative testing suites to allow implementors and operators to test response
to bugs that have been seen could also be useful.

---

From: "George, Wes" <[email-masked]>
Date: Wed, 18 Apr 2012 08:41:27 -0400

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Tony
> Li

> My recollection is that this was for static key information.  Only slightly

[WEG] That, and as we saw in both IEPG and SIDR, there are a number of potential scaling concerns about SIDR's implementation regarding both the amount of rsync sessions and the method of organizing the repository: http://www.ietf.org/mail-archive/web/sidr/current/msg04407.html

Wes George

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 18 Apr 2012 11:09:56 -0400

> [WEG] That, and as we saw in both IEPG and SIDR, there are a number of potential scaling concerns about SIDR's implementation regarding both the amount of rsync sessions and the method of organizin…

...neither of which is the 'getting the data to the router' part of
the problem though...

getting data to the router is proposed as just ssh and a bytestream or
some tcp-AO secured stream of bytes (I think).

-chris

---

From: "John G. Scudder" <[email-masked]>
Date: Wed, 18 Apr 2012 09:49:19 -0700

On Apr 17, 2012, at 3:12 PM, Tony Li wrote:

> Sure, that's better, but that's a fixed, one time multiplier on scale.

Exactly right. Just to add, the cost for the digest isn't just transmission, it's also CPU and storage. As you've already pointed out, BGP for all its warts doesn't have this particular issue.

--John

---

From: Rob Shakir <[email-masked]>
Date: Fri, 20 Apr 2012 10:00:37 +0200

> to be parsable is in fact correct?

When the error is with the logic of what is in the well-formed attributes, rather than an error with how the UPDATE itself is built, then I don't see why this assumption is dangerous. Essentially, where treat-as-withdraw is defined as the approach, then the intention is that we are confident that we can apply some form of targeted error handling to deal with this. 

A couple of examples off the top of my head:
- AS4_PATH - in this case, we had AS_CONFED_SET in AS4_PATH - the whole UPDATE is completely valid just what was in a particular attribute is not allowed.
- AS0 in AGGREGATOR (http://www.ietf.org/mail-archive/web/idr/current/msg05816.html) - again, a valid UPDATE, just where a particular set of content was not considered valid.

As I see it, we have to balance the two approaches of absolute confidence and correctness against the robustness requirements for how the protocol is deployed, and this is what is described in the draft at some length. In some deployments the requirement might be for 100% correctness, and hence perhaps the requirements described within the draft won't be desirable. However, bear in mind that the requirements in the draft are not defining that one must deploy all (or even any of) the error handling mechanisms in all deployments, but rather it states the requirements that occur when one removes the statement that you MUST send a NOTIFICATION and hence impact all NLRI. As with all compromises, it will be up to individual operators to choose the right balance for their deployment. 

The "semantic" category of errors are those where we consider that we can extract the NLRI with a degree of confidence that it is correct. If there are problems with the definition in the draft, or the types of errors that are listed - let's please discuss this here. I'd really welcome as much feedback as possible from the WG on this!

Kind regards,
r.

---

From: Tony Li <[email-masked]>
Date: Fri, 20 Apr 2012 09:27:21 -0700

> The "semantic" category of errors are those where we consider that we can extract the NLRI with a degree of confidence that it is correct. If there are problems with the definition in the draft, or…

Rob,

Thank you, I think that this was enormously constructive.

Yes, if we're willing to dive down a layer, we can classify errors as syntactic and semantic.  Syntactic would be ones that include inconsistent TLV lengths, unknown mandatory attributes, etc.  Semantic errors would be about TLV contents.

I could see taking drastic action for syntactic errors, but being more genteel about semantic errors with the exception of NLRI.  If those are busted in some way, I don't see how you can even "treat as withdraw".

Regards,
Tony

---

From: "Ilya Varlashkin" <[email-masked]>
Date: Tue, 24 Apr 2012 09:41:47 +0200

Hi Toni,

> Yes, if we're willing to dive down a layer, we can classify errors as
syntactic
> and semantic.  Syntactic would be ones that include inconsistent TLV
lengths,
> unknown mandatory attributes, etc.  Semantic errors would be about TLV
> contents.
> 

How would you classify properly formed TLV out of context? E.g. an attribute
specific to L3VPN attached to a prefix received on a session that supposed
to carry only AFI/SAFI 1/1 - such things did happen in the wild not too long
ago.

/iLya

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 24 Apr 2012 03:55:03 -0400

What matters is what to do about it.
Again:
If you can isolate it to a set of NLRI, withdraw them
(and log an error and send an operational message once that becomes available)
Else reset the session.

The goal is to isolate and report the error, not to recover from the error.

--
Jakob Heitz.

---

From: Tony Li <[email-masked]>
Date: Tue, 24 Apr 2012 08:05:28 -0700

> to carry only AFI/SAFI 1/1 - such things did happen in the wild not too long

Syntactic errors would imply that things are structurally malformed: inconsistent lengths in TLVs, for example.  Something that's "properly formed" is probably going to always be syntactically correct, so I would expect that this would fall into the semantically incorrect bucket.

Tony