Subject: Re: [GROW] [Idr] I-D Action: draft-ietf-grow-ops-reqs-for-bgp-error-handling-06.txt

From: "Chris Hall" <[email-masked]>
Date: Fri, 4 Jan 2013 23:44:09 -0000

Tony Li wrote (on Fri 04-Jan-2013 at 17:02 +0000):
> To: Jeff Wheeler
....
> I agree that the Message Length will either be correct or it won't.
> Now, all us need is an oracle (the computer science kind ;-), to
> tell us which is which.  Got any handy?  ;-)

The outermost framing for the message is the Marker (16 x 0xFF)
followed by two octets of Message Length and one of Type.  Message
Length must be > 19 <= 4096 (currently), and >= 23 for an UPDATE
message.  So, there are some constraints there.

The next level of framing is the Withdrawn Routes Length and the Total
Path Attributes Length.  And 23 + those must equal the Message Length.

That (as per the drafts) is about it, oracle-wise.  A CRC would be
nice -- but not amongst the Attributes, except, perhaps between the
MP_XXX at the front of the Attributes and the rest.  But long runs of
0xFF are pretty rare in the body of a message, so the Marker is not
bad.

When we start considering what buggy software might be capable of
throwing out, I think we are in danger of running,
Wile-E.-Coyote-wise, out into mid-air...

Most code is tested before release.  Some parts of the code are in
constant use.  I think it is practical to consider some things
dependable.  (Never 100% guaranteed, but many-sigma.)

The outermost framing for a BGP Message is common to all messages.
The code which writes the Message Length can be written so that the
value is very closely tied to the value used to dispatch the message
to the output buffers.  This stuff is pretty well exercised, and I
think that a systematic error here is unlikely.  I can imagine some
memory management, or multi-threading, or random corruption, or other
exotic bug which could affect this outermost framing.  And such a bug
could lie dormant for a long time.  These sorts of issues are
notoriously capricious and hard to reproduce.  Which is a Bad Thing
but also a Good Thing, because following a session reset the bug may
never be seen again !

The inner framing of Withdrawn Routes, Path Attribute and NLRI is also
code in constant use.  Given a discrepancy between the lengths of
these and the Message Length, if pushed I would go with the Message
Length.  But, I would prefer to treat such a discrepancy as a
session-reset.  Mostly because I think that the balance of
probabilities is that an error here is more likely to be a symptom of
some exotic error than anything else.

[Plenty of hostages to fortune there... I look forward to hearing of
counter examples :-)]

This is all per the drafts.

When it comes down to it, the most likely place for systematic errors
to occur is in obscure corners of attribute handling.  eg the
well-known RIPE/DUKE "experiment".  I think that concentrating on
those would be the most productive.  There are enough issues to
resolve there to keep one busy.

It occurs to me: one issue with treating the Marker etc. as the one
true message boundary, is that if the code finds itself reading
forwards, looking for the next start of message, that implies that the
Message Length on the _previous_ one was probably wrong...  but all
decisions relating to that previous message have already been made :-(
That's not a problem if you reset the session.  In extremis I guess
one has bigger issues to worry about !

An option to resync on Marker etc, as a last-ditch,
keep-the-session-up-at-all-costs measure, doesn't seem that difficult
to me -- and wouldn't (key consideration) interfere with code for
normal running.

Chris

---

From: Jakob Heitz <[email-masked]>
Date: Sat, 5 Jan 2013 00:00:23 +0000

Current usages of BGP probably will not have anything matching
a marker in the byte stream other than the marker.

But future additions could well do that.
There is a draft to increase the maximum length,
so you can't rely on that being <= 4096.
What about a notification message or a possible future
operational message that replays the errant update.
That will look just like an update to the resync code.

-- 
Jakob Heitz.

---

From: Jakob Heitz <[email-masked]>
Date: Sat, 5 Jan 2013 00:49:33 +0000

On Friday, January 04, 2013 4:35 PM, John Leslie <mailto:[email-masked]> wrote:

>    Definitely a bad idea!
>    Replay of an errant update can't help, and would likely crash its
> receiver. Don't do that! 

Sorry, that's not what I meant.
I meant a future operational message that says
"this update message caused me a problem and here is
the update message for your reference". It is a copy of
the message, not actually a replay.

-- 
Jakob Heitz.

---

From: John Leslie <[email-masked]>
Date: Fri, 4 Jan 2013 19:34:32 -0500

> a marker in the byte stream other than the marker.

   I can't think of any way for it to get there that isn't an error...

> But future additions could well do that.

   Sounds like a very bad idea to me.

> so you can't rely on that being <= 4096.

   Indeed: it will probably grow past 4096 sometime.

   Definitely a bad idea!

> That will look just like an update to the resync code.

   I'm not saying resync is necessarily a good thing -- but I believe
a marker (16 bytes all ones) in the middle of _any_ message deserves
an error of some kind.

   Lacking a TLV structure for messages, we should not let go of a
marker which separates messages.

--
John Leslie <[email-masked]>

---

From: Jeff Wheeler <[email-masked]>
Date: Sat, 5 Jan 2013 00:29:45 -0500

> marker which separates messages.

Even if you were inventing "BGP 5" and you cleaned up all the messy
extensions and implied field lengths, I think you would be persuaded
to keep MARKER around as a good "dummy check" on the message stream.
Historically, protocol designers have often conserved octets because
it seems like a good way to keep protocol overhead low.  For
control-plane protocols whose encoding/transmission overhead is
insignificant, this has sometimes produced poor results.

> the message, not actually a replay.

I liked your idea the first time I read it, and I continue to like it.
 This would be VERY helpful for troubleshooting.

We had some discussion before about what would happen if the original
message, being echoed back in a "this was broken" for diagnostic
purposes, could not fit into the maximum message length for the
session.

What if you didn't include the original Message header, the MARKER,
Message Type and Length?  Your "broken message" echo Message would
then be exactly the same Length as the bad one so it would definitely
fit in the maximum length, and you don't need to tell the other side
what the original Length was because it will know by the Length of
your echo Message.

As long as it is only for "broken UPDATE" and not for other kinds of
messages, then you don't need to encode the Message Type field.

The only downside is it wouldn't work for anything except a broken
UPDATE, and it would need to be extended in the future if new types of
messages are added to BGP.  These seem like reasonable limitations
though.

> to me -- and wouldn't (key consideration) interfere with code for

I don't think it's that difficult either, but I do not know if the
cost of implementing it is worth the reward.

Attribute errors are things that, historically, we've seen propagate
through the DFZ (and they might inside a datacenter network too) and
the cost of dealing with them is probably going to be worthwhile.  A
message length error isn't going to cascade through anybody's network
right now, let alone cross domain boundaries; so is it worth doing?

I only brought up re-sync because it is possible to do and it might
have some limited value if we had Jakob's "i got this bad message,
here it is" feature.  I'm still not necessarily advocating the re-sync
idea.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Jakob Heitz <[email-masked]>
Date: Sat, 5 Jan 2013 05:50:07 +0000

On Friday, January 04, 2013 9:30 PM, Jeff Wheeler <mailto:[email-masked]> wrote:

> Jakob's "i got this bad message, here it is" feature.

Just to set the record straight, it's not my idea and
I was referring to the older discussion about this.

I'll note that 
http://tools.ietf.org/html/draft-ietf-idr-operational-message-00
has expired. It would be a good complement to
http://tools.ietf.org/html/draft-ietf-idr-error-handling-03
if it were revived.

-- 
Jakob Heitz.

---

From: John Leslie <[email-masked]>
Date: Sat, 5 Jan 2013 06:50:49 -0500

> the message, not actually a replay.

   Danger, Will Robinson!

   Actually, we could invent a message type which reflects the content
of a "bad" update, but it would only be useful for logging at the
sending peer for human perusal; and, to be useful to that human it
would need to classify the problem and indicate what part(s) of the
update have been acted upon.

   Under no circumstances should we do this without making its length
abundantly clear -- both the length we assumed when reading it and the
length of what we are returning (which need not be the same: humans
get the best value out of the first few hundred bytes in most cases).
In particular, if we detect 16-bytes of marker within it, we MUST NOT
return that marker un-escaped.

   This would probably be the most helpful when we _don't_ expect to
be able to continue without a NOTIFY. We would need to send this only
to a peer that advertised the capability to log it; and that might
include the ability to recognize it _after_ the NOTIFY if we decide
that sending it before the NOTIFY is too risky...

---

From: Rob Shakir <[email-masked]>
Date: Sat, 5 Jan 2013 13:31:44 +0000

Hi Jakob,

We (the authors) are working on a re-spin of this draft. My aim is to do this within the next week or so.

Kind regards,
r.

On 5 Jan 2013, at 05:50, Jakob Heitz wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: Jeff Wheeler <[email-masked]>
Date: Sat, 5 Jan 2013 12:26:04 -0500

> update have been acted upon.

I don't think the message necessarily has to describe the problem, but
BGP already has some mechanisms for this.

If a receiver detects a malformed attribute, it will send a
Notification with the bad attribute in the Data part of the notice.
So the existing Notification will at least tell you about that.

Imagine if you are an ISP and your customer's router is complaining of
bad messages and resetting the session.  His logging is not good so
you don't have very much information to go on.  Maybe you get a
malformed attribute notice but your routers don't have any way for you
to search for what paths carry that attribute.  You will be debugging
for a while!

If the customer's router sent you the whole message you could
troubleshoot the problem more quickly even if the customer router
doesn't tell you why it choked on the message or produce any useful
log entries for the customer tech.

> return that marker un-escaped.

I agree that returning the un-escaped MARKER could have unintended consequences!

I think the whole message is useful ESPECIALLY if the NLRI are moved
to the beginning of messages at some time in the future.  After all,
if they are moved, then attributes will be at the end of the message,
so you might need them.

If the "oh no" message is literally the bad message with a different
Message Type, then you will get everything including what the receiver
thought the length was.  The only things you will not get is the
original marker and message type.  If you only generate "oh no" if the
marker is believed to be legit and the Message Type was UPDATE, then
these are implicit whenever you get an "oh no" message.

> that sending it before the NOTIFY is too risky...

Actually if error-handling introduces the ability to treat-as-withdraw
or ignore-bad-message then it seems like it will become useful to have
this new diagnostic message even if the BGP session is not reset.
That way both sides of the session have some diagnostic information
about problems that are potentially being masked by the use of
error-handling.