Subject: Re: [Idr] draft-ymbk-bgp-extended-messages-01

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 10:34:35 -0700

the following text has been added to sec 3 of the unpublished -02

   Applications generating messages which might be encapsulated within
   BGP messages MUST be aware of the maximum message size and all
   encapsulation overheads on the path the encapsulated data are
   expected to traverse.

randy

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 22 Mar 2011 18:54:40 +0100

>    encapsulation overheads on the path the encapsulated data are

What does the "path the encapsulated data are expected to traverse" matter
here? BGP is still using TCP and TCP is a byte stream. There is no
encapsulation overheads other then the BGP message encodeing itself.

Do we really need to make everything much more complex than it actually
is?
-- 
:wq Claudio

---

From: Keyur Patel <[email-masked]>
Date: Tue, 22 Mar 2011 10:59:30 -0700

Claudio,

> encapsulation overheads other then the BGP message encodeing itself.

We have Notification and Diagnostic messages that could contain a BGP update
message. 

Regards,
Keyur

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 11:10:02 -0700

> What does the "path the encapsulated data are expected to traverse"
> matter here?

i expect to see suggestions of mpls label stacks within bgp that are
expected to traverse multiple bgp speakers on this list any day now.
this wording anticipates them.

> BGP is still using TCP and TCP is a byte stream. There is no

bgp does not encode itself.  some people have an application of a bgp
hack that then encapsulates a bgp message.  all this change says is the
blatantly obvious (one would have thought), if you think your sour cream
might be encapsulated in a bgp message, then use less sour cream.

> Do we really need to make everything much more complex than it actually

hey, i did not come up with this encapsulation stuff.  i am just trying
not to let every flavor of dairy causing changes in the basic message.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Mar 2011 21:22:02 +0100

Hi Randy,

The text below does not reflect the proposal from Enke nor does it fix 
the issue.

This is not the point for application to be aware of max BGP message 
size. This is about allowing one BGP message from SAFI X to contain 
entire BGP message from SAFI Y.

If you make SAFI Y massage as big as it is possible with two octet 
length field it effectively means that SAFI X may include it only by 
cutting it into two pieces.

And how do you cut malformed update which you can not parse in the first 
place ? Then you need to worry about reassemble on the other side. Sure 
everything is possible, but I think we should choose the easiest 
solutions. Reducing max BGP message size of just few octets to allow 
extra header seems quite a reasonable suggestion.

Many thx,
R.

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Mar 2011 21:30:35 +0100

Hi Randy,

> might be encapsulated in a bgp message, then use less sour cream.

I agree with this very much. But this is not about some other 
applications proposing to carry Eiffel Tower of MPLS label bricks in BGP 
nor put Brittanica inside one BGP message. It would be blatantly obvious 
indeed that such application better handle it - it has no choice anyway.

The problem is different as described in the other mail.

Many thx,
R.

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 13:29:39 -0700

> cutting it into two pieces.

then don't do that.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Mar 2011 21:33:41 +0100

We have IP operators asking to send them back malformed update within 
BGP (no new protocol to establish between ASes).

Should the answer be: Sorry dear ISP - No - since Randy said so ?

I am personnaly fine either way .. let's just make it clearly documented 
on the list and move on.

R.

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 13:41:24 -0700

> Should the answer be: Sorry dear ISP - No - since Randy said so ?

i am beginning to suspect that you place a different value on time than
i.

if you want to encaps, that which is encapsulated will have to be
smaller than the transport limit.  what you propose to to is pretend the
transport limit is the limit-<encaps overhead>, which is just a play on
words.

if you want to be able to encaps something later, make it small enough
to fit.  you can rename it all you want, but you ain't gonna escape it.

randy

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 22 Mar 2011 22:11:31 +0100

> this wording anticipates them.

But that's the same with AS_PATH attributes or extended communities. Yes,
if you plan to prepend 1024 times your AS then you will hit a brick wall.
Nothing new here.

> might be encapsulated in a bgp message, then use less sour cream.

Yes, realized that I used the wrong word. s/encoding/encapsulating/
In the end it should be obvious that the sum of attributes must stay below
the bgp message limit. I just don't see why there is this fuzzy notion of
overheads ... expected to traverse. What should an implementor do with
such unspecific informations? In which case would I need to reduce the
message size to leave room for additional overhead and how much is needed?

> not to let every flavor of dairy causing changes in the basic message.

I guess it is the duty of the dairy queen to ensure that all flavors fit
into a bgp message (be it 4k or 64k).

-- 
:wq Claudio

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 14:15:02 -0700

> I guess it is the duty of the dairy queen to ensure that all flavors
> fit into a bgp message (be it 4k or 64k).

bingo!

and, if you think the person with whom you are sharing the cone might
add chocolate, nuts, whipped cream, a cherry, ...  then you can guess
how much less to put in.  i don't want to guess.  i think this is all
way too kinky.  i am from the pacific northwest, not complexifornica.

randy

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 22 Mar 2011 22:23:24 +0100

On Tue, Mar 22, 2011 at 09:33:41PM +0100, Robert Raszuk wrote:
> 
> We have IP operators asking to send them back malformed update
> within BGP (no new protocol to establish between ASes).
> 

No the answer is, we try our best but maybe part of the message is
truncated because it exceeds the message boundary. This is needed anyway
since the proposed Extended Message support is asymetric.
It is a diagnostic message and so truncation is acceptable.

> I am personnaly fine either way .. let's just make it clearly
> documented on the list and move on.
> 
> R.
> 

Actually I don't see how this would be possible with the current MP spec
(at least the NLRI encoding is limited to 256bits and so way smaller then
the current limit).

-- 
:wq Claudio

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Mar 2011 22:27:06 +0100

Hi Claudio,

> (at least the NLRI encoding is limited to 256bits and so way smaller then

I do not put it into MP_REACH :)

See http://tools.ietf.org/html/draft-raszuk-bgp-diagnostic-message-02 
for encoding details of new message format.

Many thx,
R.

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 22 Mar 2011 23:50:50 +0100

> It is a diagnostic message and so truncation is acceptable.

I don't share view that truncation is acceptable. As an operator I would like to see whole message that caused the problem. And my understanding of current situation regarding message size is that coder of non-diagnostic message has two choices:

a) feel free to send MAX_SIZEd message and expect that other side will just truncate, or
b) be aware that his/her message can be encapsulated inside other message at some point and therefore he/her MUST leave room for those possible encapsulations

and at the moment going either way will be compliant with specification. So we may end up with some implementations in the field that don't leave room for secondary encapsulation and some that do, and some that do it by a switch. With operator's hat on I would rather have specification clearly saying that we want option (b) in all compliant implementations.

If we can agree on the above, then second issue to resolve is which messages are allowed to be MAX_SIZEd and which should allow room for other encapsulations. And finally we need to agree how much room exactly to leave.

Kind regards,
iLya

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 22 Mar 2011 19:13:01 -0400

I agree with Ilya.

It's not going to hurt anyone to shave a hundred or two
bytes off the end.

I don't even like the maximum size of 65535.

No concrete reason, I just feel nervous pushing
things to the red line.

--
Jakob Heitz.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Ilya Varlashkin
> Sent: Tuesday, March 22, 2011 3:51 PM
> To: IETF IDR; Claudio Jeker
> Subject: Re: [Idr] draft-ymbk-bgp-extended-messages-01
> 
> 
> I don't share view that truncation is acceptable. As an 
> operator I would like to see whole message that caused the 
> problem. And my understanding of current situation regarding 
> message size is that coder of non-diagnostic message has two choices:
> 
> b) be aware that his/her message can be encapsulated inside 
> other message at some point and therefore he/her MUST leave 
> room for those possible encapsulations
> 
> and at the moment going either way will be compliant with 
> specification. So we may end up with some implementations in 
> the field that don't leave room for secondary encapsulation 
> and some that do, and some that do it by a switch. With 
> operator's hat on I would rather have specification clearly 
> saying that we want option (b) in all compliant implementations.
> 
> If we can agree on the above, then second issue to resolve is 
> which messages are allowed to be MAX_SIZEd and which should 
> allow room for other encapsulations. And finally we need to 
> agree how much room exactly to leave.
> 
> 
> Kind regards,
> iLya
>

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 16:29:12 -0700

> that coder of non-diagnostic message has two choices:
> 

real close, but not quite.  it is not specified what the encapsulator
who ran out of room will do.  truncate is a possibility.

> b) be aware that his/her message can be encapsulated inside other
> message at some point and therefore he/her MUST leave room for those
> possible encapsulations 

good idea

> With operator's hat on I would rather have specification clearly

well, it does say MUST.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 23 Mar 2011 00:33:32 +0100

>> saying that we want option (b) in all compliant implementations.

"MUST be aware" ... That does not define what "MUST" encoding/action 
should be. An important difference.

R.

---

From: Keyur Patel <[email-masked]>
Date: Tue, 22 Mar 2011 16:35:21 -0700

IMHO the draft should not document how many *bytes to leave*. Instead it
should be flexible enough and should be left upon implementations. As long
as an implementation is able to encapsulate BGP messages it should be good
enough.

Regards,
Keyur

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 23 Mar 2011 00:37:34 +0100

Hi Jakob,

 > I don't even like the maximum size of 65535.

That max massage size resulting in taking all bits is just not 
acceptable to me. This value should remain reserved till someone comes 
with new draft where this length indicates a special BGP message handling.

So at most draft-ymbk-bgp-extended-messages could use 65534.

Cheers,
R.

PS. On the other topic I do agree with Ilya that allowing 
implementations to choose what max message size of BGP UPDATE Message 
should be for a given SAFI under set of previously negotiated 
capabilities is just a pure receipt for problems. How would we handle 
dynamic capabilities ? Would that automatically result in essentially 
per peer different max BGP UPDATE Message size limit ? Nightmare to 
troubleshoot - that's what IMHO it will be.

---

From: Ilya Varlashkin <[email-masked]>
Date: Wed, 23 Mar 2011 00:41:00 +0100

On Mar 23, 2011, at 24:33 , Robert Raszuk wrote:

> "MUST be aware" ... That does not define what "MUST" encoding/action should be. An important difference.
> 

ehrm...I've actually said "MUST leave room". To summarize, how about adding following text into the draft:

"As some BGP messages potentially can potentially be encapsulated inside other BGP messages, implementations compliant to this specification MUST limit size of messages of Type X1, ... Xn to MAX_SIZE - MAX_OVERHEAD"

?

/iLya

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 16:41:18 -0700

ok

   Applications generating messages which might be encapsulated within
   BGP messages MUST limit the size of their payload to take into
   account the maximum message size and all encapsulation overheads on
   the path the encapsulated data are expected to traverse.

i feel sorry for the payloaders trying to find out the pmtu

randy

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 23 Mar 2011 00:43:08 +0100

On Tue, Mar 22, 2011 at 10:27:06PM +0100, Robert Raszuk wrote:
> See
> http://tools.ietf.org/html/draft-raszuk-bgp-diagnostic-message-02

You mentioned putting bgp messages of SAFI X in SAFI Y. Diagnostic
messages use a new message type but they have nothing todo with SAFIs.

On Tue, Mar 22, 2011 at 09:22:02PM +0100, Robert Raszuk wrote:
...
> cutting it into two pieces.

So this has nothing todo with SAFI but with the idea to be able to put a
full BGP message for diagnostic reasons into another BGP message.
Now I'm a bit less confused.

-- 
:wq Claudio

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 22 Mar 2011 19:44:26 -0400

How about some numbers:
maximm message size = 65000
maximum update message size = 64000

It won't hurt anyone to have it a little short of 65535
and it gives us a little playroom for the "what if" cases.

--
Jakob Heitz.

> From: Keyur Patel [mailto:[email-masked]] 
> Sent: Tuesday, March 22, 2011 4:35 PM
> To: Jakob Heitz; Ilya Varlashkin; IETF IDR

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 23 Mar 2011 00:48:06 +0100

Hi Claudio,

> messages use a new message type but they have nothing todo with SAFIs.

Yes indeed .. it was just for the purpose of descriptive illustration of 
the problem. The problem being the max BGP message size itself not 
related to any set of SAFIs.

Rgs,
R.

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 22 Mar 2011 19:50:50 -0400

I didn't think of large refresh or open message...
maximum notification message size = 65000
maximum non-notification message size = 64000

--
Jakob Heitz.

> Behalf Of Jakob Heitz
> Sent: Tuesday, March 22, 2011 4:44 PM
> To: Keyur Patel; Ilya Varlashkin; IETF IDR

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 16:52:44 -0700

> and it gives us a little playroom for the "what if" cases.

throwing kitty litter over your off by onw errors is a giggle.  trying
to cover up an off by 1000 error had me laughing for five minutes.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 23 Mar 2011 00:53:27 +0100

Hi Jakob,

> and it gives us a little playroom for the "what if" cases.

It would be so clean and simple ... I 100% agree with your proposal. To 
make max BGP Message size of X and max BGP Update Msg of Y where Y < X. 
We could talk what the Y and X should be.

Cheers,
R.

---

From: Jeff Wheeler <[email-masked]>
Date: Tue, 22 Mar 2011 19:54:51 -0400

On Tue, Mar 22, 2011 at 6:50 PM, Ilya Varlashkin <[email-masked]> wrote:
> I don't share view that truncation is acceptable. As an operator I would like to see whole message that caused the problem.

So you want the neighbor to encapsulate a problematic message within
another message that indicates a problem.  This is not a bad thing to
want, but there are only three ways to get there:
1) define two different maximum message sizes, one for "signaling" and
one for "diagnostic"
2) change the header format to indicate "signaling" disposition vs
"diagnostic" disposition and don't encapsulate at all
3) fragment/reassemble

Even if you decide to reduce the proposed maximum message size from
65535 to 64000 this will not change the fact that an erroneous
signaling message which is 64000 bytes long could not be encapsulated
into a diagnostic message without violating the maximum message
length.  You can't "have your cake and eat it, too."

I suggest you pick some of these possible options and argue your
position.  My opinion is that the possibility of truncated diagnostic
messages is better than any of the three alternatives above.  If I had
to choose one, I'd pick #1, above, which is the least complex to
implement.  The only reason this looks like a kludge is because
anything will be unless the header format is modified, which is
clearly not desirable.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 23 Mar 2011 01:01:26 +0100

> maximum update message size = 64000

These two need to be maximum recv size and an maximum send size (unless it is
one of the special BGP messages that may carry another message). 
The maximum send size is not a hard limit but more of a line that should
not be crossed when generating messages. Still this does not protect from
encapsulation issues. So the important question to solve is what happens
when an update exceeds the maximum message size (be it 4096, or any other
new number)? IIRC neither RFC 4271 nor any other RFC define a behaviour
for this case.

-- 
:wq Claudio

---

From: Ilya Varlashkin <[email-masked]>
Date: Wed, 23 Mar 2011 01:09:53 +0100

> "diagnostic" disposition and don't encapsulate at all

I prefer option (1) and got feeling that this is what we're discussing, not merely reducing general max_size. With opt 2 & 3 there is danger that single message/fragment is all what's going to reach other end while TCP connection is still open.

The reason why I would like whole messages is because I've seen messages that caused session reset, but the part that was sent inside notification looked perfectly fine. And from Robert's hint looks like there are other operators that share same view.

/iLya

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 17:12:05 -0700

i suggest that any hard specified limits, extra bytes, ... you want to
diddle around 64k also be applied retroactively to 4k.  clearly all the
same so-called logic should apply.

and maybe you will find it easier to think of all this as the draft
having a secret limit of 128k, but the 64k was whittled down to allow
all the cruft you seem to want to invent. :)

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 17:28:00 -0700

> I prefer option (1) and got feeling that this is what we're
> discussing

but one would think you would discuss it in the designs and drafts which
differentiate between signaling and diagnostics.

this draft does not can can not, as it can not predict all the kink the
compexifornicans will come up with in the next N years.  and "oh, let's
throw 592.4 bytes at it, surely it will be enough," philosophy should be
embarrassing.  any hard coded fudge will turn out to be wrong.

randy

---

From: Jeff Wheeler <[email-masked]>
Date: Tue, 22 Mar 2011 20:36:11 -0400

On Tue, Mar 22, 2011 at 8:09 PM, Ilya Varlashkin <[email-masked]> wrote:
> The reason why I would like whole messages is because I've seen messages that caused session reset, but the part that was sent inside notification looked perfectly fine. And from Robert's hint looks like there are other operators that share same

I can buy into this argument.  If this is the general consensus, the
remaining question is, what should the maximum message size(s) be?

I have no gripes with Jakob Heitz's numbers, 64K for "signaling" and
65K for "diagnostic."  You could ask some questions:

Question #1: why 64000 and not 65535?  Why not 64000?

My admittedly off-the-cuff understanding of the issue is that growing
the maximum message size from 4096 to any of these values is an
order-of-magnitude change, and if that isn't enough, then people
should be advocating for more than 65535, not for the relatively
inconsequential difference between 65535 and 64000.  Like Randy says,
maybe it could have been 128K; but unless someone is arguing in favor
of a much bigger size, 64000 should suit pretty much anyone.

IMO there should be no implementation difficulty which arises for a
64000 byte maximum size, and does not also arise for a 65535 maximum
size.

Question #2: why a difference of 1000 bytes for diagnostic
encapsulation / "outer message?"

Again, off-the-cuff, I think this could be as small as 19(?) bytes
permitting only a single "outer message" for encapsulation.  However,
picking the minimum possible value would guarantee that this
difference would have to change in the eventuality that the message
header ever grows.  Once again, if that 1000 bytes worth of extra
payload space really matters to any application, then that application
needs an advocate who is asking for more than one order of magnitude
increase, i.e. they should not be any more satisfied with 65535 than
they are with 64000, and the difference between 19 bytes and 1000
bytes is inconsequential.

You folks will have to excuse me for writing "bytes" instead of
"octets" or using the term "diagnostic message" where there may be a
more appropriate one already in common use.  Please feel free to read
what I mean, not what I say.

> embarrassing.  any hard coded fudge will turn out to be wrong.

I don't understand what your position is on this subject.  Perhaps you
could sum it up into favoring one or more of the proposals or opposing
all of them in favor of truncation or message header modifications, or
some other scheme you have in mind?

---

From: Ilya Varlashkin <[email-masked]>
Date: Wed, 23 Mar 2011 01:44:07 +0100

After brief off-the-list conversation I agree with Keyur and with Randy regarding generalizing only on "MUST leave room for encapsulating this message inside a message that can ever come from _direct peer_" (i.e. without specifying by how much less certain messages must be). To add context: Speaker A has negotiated some set of Capability Codes with Speaker B and therefore it's granted that Speaker A knows what Speaker B can potentially send, so Speaker A can calculate how much room to leave. Such approach paves the way for any further protocol enhancements while satisfying basic operators' need - to ensure that diag messages will contain complete version of the message that caused problem.

/iLya

/iLya

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 21:00:02 -0700

>> The reason why I would like whole messages is because I've seen
>> messages that caused session reset, but the part that was sent inside
>> notification looked perfectly fine. And from Robert's hint looks like
>> there are other operators that share same

no.  the question is to what size a bgp sender interested in a whole
error response should limit it's outgoing message given that the max
response message size will be 4k or 64k.  the answer is that this draft
should not care.  

it is the documentation of the encapsulation which needs to say, "if you
want my whole response, then limit your outgoing message size to the
agreed max, 4k or 64k, less the encaps overhead which is 19.3, or
whatever.

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 22 Mar 2011 21:01:08 -0700

> After brief off-the-list conversation I agree with Keyur and with
> Randy regarding generalizing only on "MUST leave room for
> encapsulating this message inside a message that can ever come from
> _direct peer_" (i.e. without specifying by how much less certain
> messages must be). To add context: Speaker A has negotiated some set
> of Capability Codes with Speaker B and therefore it's granted that
> Speaker A knows what Speaker B can potentially send, so Speaker A can
> calculate how much room to leave. Such approach paves the way for any
> further protocol enhancements while satisfying basic operators' need -
> to ensure that diag messages will contain complete version of the
> message that caused problem.

yes.  precisely.

randy

---

From: Jeff Wheeler <[email-masked]>
Date: Wed, 23 Mar 2011 02:36:30 -0400

> agreed max, 4k or 64k, less the encaps overhead which is 19.3, or

Is your position then that you have more confidence in vendors
correctly implementing the above behavior than in same vendors dealing
with two distinct maximum message sizes, one for "signaling" and one
for "diagnostic?"

I still have "off by twenty" BGP-related bugs that I've reported to
vendors many years ago, which have not and probably will not be fixed.
 My opinion is that two distinct sizes is more likely to be correctly
implemented without introducing new bugs.  Instead of vendors' BGP
implementations correctly reducing the maximum "signaling message"
size to account for encapsulation of same messages within a diagnostic
outer-message header, the kinds of related bugs I foresee are:
1) failing to reduce signaling message size
1a) resulting in truncation of related diagnostic messages; or
1b) resulting in too-long diagnostic messages which violate the single
maximum message size
2) incorrectly reducing signaling message size too much
3) incorrectly applying the reduced signaling message size to
diagnostic messages

I think we disagree on a basic issue here.  If I were Ilya, and cared
a lot about receiving the whole offending message in case of trouble,
I would want there to be few obstacles standing in the way of vendors
getting that right.

In both cases, we're really talking about the same thing -- two
different maximum message sizes.  Yours comes up with two different
sizes by way of calculating the exact encapsulation overhead needed
and reducing signaling message size as appropriate.  Jakob Heitz'
notion simply spells out two distinct constants.

---

From: Randy Bush <[email-masked]>
Date: Wed, 23 Mar 2011 00:50:35 -0700

> with two distinct maximum message sizes, one for "signaling" and one

i have far more confidence in vendors correctly implementing a simple
and clear spec than a shotgun crap-shoot of second-guessing and wild
estimation.

this week it's diagnostics.  next meeting it will be bgp wrapped mpls
encapsulated dns rrs of ssh keys of routers.  it makes zero sense to rev
this document for every allegedly bright idea that comes down the idr
septic pipe.  this document is a simple statement of pdu max.  folk who
want to do exciting things can keep the excitement in their documents.

randy

---

From: Jeff Wheeler <[email-masked]>
Date: Wed, 23 Mar 2011 07:11:36 -0400

> and clear spec than a shotgun crap-shoot of second-guessing and wild

If this is the case, then your spec should be simple and clear, and
>encapsulation overheads on the path the encapsulated data are

As has been demonstrated on this thread, "all encapsulation overheads
on the path the encapsulated data are expected to traverse" may lead
some implementing persons to believe that means it must account for
transport (TCP) overhead, which is unrelated to the maximum size of
the BGP message.

Perhaps you should revise your proposal to state:
=Applications generating messages which are nested within BGP
outer-messages, such as diagnostic notifications, must be aware that
the maximum message size applies to the outer-most BGP message.
Applications relying on a BGP neighbor's ability to encapsulate a
transmitted message within an outer-header and echo it back must
further reduce the maximum size to account for the outer-header size
and any additional data payload.

Two different message sizes is also quite simple and clear.  If you're
having trouble wrapping your head around this, I suggest you think
about it from the perspective of the programmer implementing BGP on
his employer's router.  Standards documents must not be goals in and
of themselves; they should be tools for getting from "feature
customers want" to "bullet point on vendor datasheet" using a common
and inter-operable mechanism.

How will Mr Programmer decide to implement Ilya's desire to receive
whole-message diagnostic messages in response to bad/unhandled
signaling messages?  Probably, he's going to pass this message to a
distinct subroutine that wraps it up in the outer-message and sticks
it in the output queue.  This subroutine is going to have to do a
bounds check against the maximum payload size it is allowed to send to
the neighbor.  It can either check against a computed payload size
which the vendor will probably get right, because it's not that hard
to subtract 19 from the maximum message size for the neighbor; or it
can check against a constant that has less opportunity for mistakes
and some "wiggle room" for catching cases where the offending neighbor
has accidentally sent a "too big" message and reporting that to said
neighbor.

You say this is a "crap-shoot of second-guessing."  I say it's quite
reasonable to draw a distinction between signalling and
notification/diagnostic messages, and to include in the diagnostic
process a buffer against bugs.  After all, the diagnostic message Ilya
wants will very likely be in response to bugs on one side or the
other, be they capabilities negotiation / interpretation failures,
off-by-ones (or twenties), or whatever.

> want to do exciting things can keep the excitement in their documents.

There must be someone who appreciates your larger-than-life examples
of why this is some kind of slippery slope.  I don't, and I take these
to mean that you do not have a realistic and practical example of how
this is somehow unwise, dangerous, or more difficult for vendors to
implement, either now or in a future which doesn't require suspending
disbelief.

If you want to define what are supposed to be industry-standards based
on an irrational fear of wrapping some "mpls encapsulated dns rrs of
ssh keys of routers" inside BGP messages, I suggest you work harder on
getting "The Social Network 2: The Standards-Track" green-lit, because
that kind of imaginative but utterly baseless argument is not
beneficial to the operators or the vendors.  The fact that
"contributions" like your outlandish example sometimes drive the
standards process is deplorable.

---

From: Paul Jakma <[email-masked]>
Date: Wed, 23 Mar 2011 11:13:52 +0000 (GMT)

On Tue, 22 Mar 2011, Ilya Varlashkin wrote:

> If we can agree on the above, then second issue to resolve is which 
> messages are allowed to be MAX_SIZEd and which should allow room 
> for other encapsulations. And finally we need to agree how much 
> room exactly to leave.

The core problem is having a limit in the underlying transport (which 
happens to be the same protocol). Of course it's quite possible to 
remove limits (use a length field with a width large enough to make 
the problem go away effectively for ever, or instead remove the need 
for pre-determined length and use an end delimiter).

Alternatively, define a fragmentation capable encapsulation for BGP.

(Warning: This message has been made in a factory that handles 
dead-pan humour).

regards,
-- 
Paul Jakma  [email-masked]  twitter: @pjakma  PGP: 64A2FF6A
Fortune:
Where humor is concerned there are no standards -- no one can say what
is good or bad, although you can be sure that everyone will.
 		-- John Kenneth Galbraith

---

From: Rob Shakir <[email-masked]>
Date: Wed, 23 Mar 2011 11:31:00 +0000

> The reason why I would like whole messages is because I've seen messages that caused session reset, but the part that was sent inside notification looked perfectly fine. And from Robert's hint look…

Hi Ilya, IDR,

Just to add some further voice to this, and add my £0.02, I am an operator that cares about the above - for the same reasons that Ilya highlighted in this thread. Partial message replay in the case that an erroneous or corrupt packet is received really limits the utility of such mechanisms.

> After brief off-the-list conversation I agree with Keyur and with Randy regarding generalizing only on "MUST leave room for encapsulating this message inside a message that can ever come from _dire…

It would seem to me that this wording provides the clearest specification, and highlights the issue of message encapsulation without leaving the specification with scope for revision on each change. As Randy says, for special cases, it would seem that those extensions requiring this should handle the specific caveats of this within the relevant standard.

Kind regards,
r.

This e-mail has been scanned for viruses by the Cable&Wireless Worldwide e-mail security system. For more information on a proactive 
managed e-mail secure service, visit http://www.cw.com/managed-exchange

The information contained in this e-mail is confidential and may also be subject to legal privilege. It is intended only for the recipient(s) named above. 
If you are not named above as a recipient, you must not read, copy, disclose, forward or otherwise use the information contained in this email. If you 
have received this e-mail in error, please notify the sender (whose contact details are above) immediately by reply e-mail and delete the message and any 
attachments without retaining any copies.

Cable & Wireless Worldwide plc 
Registered in England and Wales. Company Number 07029206
Registered office: Liberty House, 76 Hammersmith Road, London W14 8UD, England

---

From: "iLya" <[email-masked]>
Date: Wed, 23 Mar 2011 12:34:53 +0100

--------------------------------------------------
> happens to be the same protocol). Of course it's quite possible to remove 
> limits (use a length field with a width large enough to make the problem 
> go away effectively for ever, or instead remove the need for 
> pre-determined length and use an end delimiter).
>

Besides the fact that fragmentation is evil, there is also third alternative 
to the above two, and that is what Keyur and Randy have brought up, and 
which after some consideration I perceive as solid and flexible. This 
alternative is to require BGP speaker to calculate maximum possible overhead 
(it _can_ do it based on agreed Capability Code set) for given peer (!) and 
use that value instead of predefined fixed size overhead. I do put exactly 
same level of trust in implementor following this requirement as I would in 
trusting implementor obeying fixed-sized overhead requirement.

Kind regards,
iLya

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 23 Mar 2011 13:12:17 +0100

On Wed, Mar 23, 2011 at 11:31:00AM +0000, Rob Shakir wrote:
> > there are other operators that share same view.
> 
> Hi Ilya, IDR,
> 
> Just to add some further voice to this, and add my £0.02, I am an
> operator that cares about the above - for the same reasons that Ilya
> highlighted in this thread. Partial message replay in the case that an
> erroneous or corrupt packet is received really limits the utility of
> such mechanisms.
> 

Just how many times did a 4096 byte long BGP message caused a notification
for you? Isn't best effort good enough in such a rare case plus the error
needs to be in the last few bytes of the update as well. As an implementor
I need to say it is probably better to accept this instead of creating a
complex system around it which is most probably more prone to fail then the
problem it should actually solve.

-- 
:wq Claudio

---

From: "iLya" <[email-masked]>
Date: Wed, 23 Mar 2011 13:23:44 +0100

> for you? Isn't best effort good enough in such a rare case plus the error

since we're talking here about extending message length far beyond original 
4K octets, there's good chance that we will start seeing more of the large 
messages in the field (not necessary in the global IPv4/IPv6 tables), and 
since solution to address rised concern seems both feasible for implementors 
and satisfactory to operators then naturally it seems like a good thing to 
stick to.

Kind regards,
iLya

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 23 Mar 2011 13:49:08 +0100

On Wed, Mar 23, 2011 at 01:23:44PM +0100, iLya wrote:
> since we're talking here about extending message length far beyond
> original 4K octets, there's good chance that we will start seeing
> more of the large messages in the field (not necessary in the global
> IPv4/IPv6 tables), and since solution to address rised concern seems
> both feasible for implementors and satisfactory to operators then
> naturally it seems like a good thing to stick to.
> 

Yes, but it is even less of an issue in the case larger messages are
available. Currently it is possible to get all information into 4k and so
from a math perspective making the maximum message size bigger will reduce
the risk of truncation even more.

Yes the 4k BGP message size is starting to cause problems but we're
talking here about a new limit that is 16 times bigger. I don't want to
know what kind of information needs to be added to make an update exceed
even 32k. I hope that somewhen people will realize that BGP is not a
kitchen sink.

-- 
:wq Claudio

---

From: Rob Shakir <[email-masked]>
Date: Wed, 23 Mar 2011 13:56:24 +0000

Hi Claudio,

> problem it should actually solve.

We already know that this problem is relatively difficult for us to solve as operators - isolating a single message that is triggering NOTIFICATION is not trivial within many live environments. The cost of not being able to solve this issue in some cases is high (where we have a high impact session that is actually generating NOTIFICATION based on this message we cause outages to subsets of NLRI). I think should keep in mind that operational visibility of any protocol is key to being able to debug it correctly, therefore I would much prefer that we avoid generating some case whereby we have cases that are harder to debug, especially those that are "edge" cases (such as at the boundaries of available lengths). As such, I therefore think that keeping the capability to support these cases is very useful.

>From your view as an implementor, I'd be very interested to hear whether it's actually likely to be that one would introduce disproportionate complexity by implementing some check as suggested by Ilya (i.e. I can receive a message of up to length X from my peer, therefore I should transmit messages only up to X-N where N is the maximum length of the 'encapsulation' of the message). From my (very naïve) view, it does not sound to exceed an acceptable level of complexity? Indeed, perhaps the most complex scenario is around where one replicates a message to multiple neighbours, but in this case, this is easily handled by selecting the minimum largest available message size for a peer.

> I hope that somewhen people will realize that BGP is not a
> kitchen sink.

This is a whole other debate ;-), which touches on scaleability, and network deployment complexity. Unfortunately, I think we should rather deal with the situation that BGP-4 provides a key signalling protocol to operators, rather than one based on aspirations away from this position.

Kind regards,
r.

---

From: David Freedman <[email-masked]>
Date: Thu, 24 Mar 2011 12:47:09 +0000

On 23/03/11 11:34, iLya wrote:
>>
> 
> Besides the fact that fragmentation is evil, there is also third
> alternative to the above two, and that is what Keyur and Randy have
> brought up, and which after some consideration I perceive as solid and
> flexible. This alternative is to require BGP speaker to calculate
> maximum possible overhead (it _can_ do it based on agreed Capability
> Code set) for given peer (!) and use that value instead of predefined
> fixed size overhead. I do put exactly same level of trust in implementor
> following this requirement as I would in trusting implementor obeying
> fixed-sized overhead requirement.

I think the capability-value should be used to define the
max-message-size, so the implementer can decide what they are happy (and
competent) to accept and process.

I also don't see a need for symmetry either.

If the situation arises where an implementation decides it really needs
to send a message its peer which is larger than the peer's
max-message-size (either through evaluation of the cap-value or lack of
cap) then it could either send a NOTIFICATION (if the message was
critical to operation) or ADVISORY (yet another use case)
with an optional summary of the message contained.

Dave.

- -- 

David Freedman
Claranet

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 24 Mar 2011 09:03:48 -0400

I hope you don't mean to put the actual value into the capability. Then I'll have 1000 different peers with 1000 different sizes. What a nightmare of code that would be, for no good reason. You will be sure to get a few malformed then. 

--
Jakob Heitz.

On Mar 24, 2011, at 5:47 AM, "David Freedman" <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iEYEARECAAYFAk2LPUwACgkQtFWeqpgEZrJa/wCfYee9U9OtHvPKr2oOPD64wYrf
> aQYAnRky55RnNZC+tQI4OBkpcCIMEcUL
> =iBTX
> -----END PGP SIGNATURE-----

---

From: David Freedman <[email-masked]>
Date: Thu, 24 Mar 2011 13:11:43 +0000

> I hope you don't mean to put the actual value into the capability. Then I'll have 1000 different peers with 1000 different sizes. What a nightmare of code that would be, for no good reason. You wil…

Why not? we do this with TCP already!

The only advantage I can possibly see for standardising on the extended
size would be if an implementation used it to generate bigger UPDATE
messages (i.e faster convergence) and wanted saner grouping for peers
for this output processing (and even then, I'm sure it could come up
with a solution based on thresholds)

Dave.

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 24 Mar 2011 14:15:30 +0100

> competent) to accept and process.

Please no. Having to cope with 4k and >4k clients is enought of a
problem. We don't need the interop issues caused by systems announcing
starange max-message-size values.
 
> I also don't see a need for symmetry either.

We do need symmetry because of the fact that notfications may need to
be sent back with more then 4096 bytes of data in them. Handling asymmetry
does only gain more complexity but nothing else. KISS - keep it simple and
stupid is the way to build robust protocols (no buttons, no unneeded fluff
or features).

> with an optional summary of the message contained.

Yes, there is a need to define the behaviour of BGP speakers when messages
exceed the maximum size but I have the feeling that this should be covered
in a different document since this also affects systems that only use 4k
buffers.

-- 
:wq Claudio

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 24 Mar 2011 14:28:52 +0100

> Why not? we do this with TCP already!

TCP is a byte stream and has therefor no message size. What is the gain of
having clients with BGP message sizes of 4096, 5000, 6000, 8000, 10000,
16000, 18000, ..., 32000 bytes? I don't want to imagine the nightmare of
having to admin such a network.
 
> with a solution based on thresholds)

Transport and message sizes are not the problem to achive faster
convergence. My systems are able to exchange the full table in a lot less
then 10 seconds (from open until all messages are processed). Do you think
this needs to be faster?

-- 
:wq Claudio

---

From: David Freedman <[email-masked]>
Date: Thu, 24 Mar 2011 15:12:36 +0000

> starange max-message-size values.

Well, there is no intention of interop burden here, you need to support
at least 4096 , you can grow above this if you use the cap , otherwise
the min size you need to interop is 4096.

> stupid is the way to build robust protocols (no buttons, no unneeded fluff

Well, if its a notify, it is session terminal, we can tear the session
down (the primary objective) without having to preserve all the message
content (and if we do have to preserve the message content, then it must
be important enough to require both peers to support these larger
messages anyway)

>TCP is a byte stream and has therefore no message size. What is the
>gain of
>16000, 18000, ..., 32000 bytes? I don't want to imagine the nightmare >of

Not sure what kind of additional admin work you may need to do, if
implementers are confident enough to standardise on always trying to
negotiate this cap with 65535 then so be it.

>Transport and message sizes are not the problem to achieve faster
>convergence. My systems are able to exchange the full table in a lot >less
>then 10 seconds (from open until all messages are processed). Do you >think
>this needs to be faster?

The "full internet routing table" example take less time to transport
with larger TCP MSS and BGP message sizes (for the UPDATE)

Of course, retransmitting these large messages (if they are
lost/corrupted) is another issue!

- -- 

David Freedman
Group Network Engineering

[email-masked]
Tel +44 (0) 20 7685 8000

Claranet Group
21 Southampton Row
London - WC1B 5HA - UK
http://www.claranet.com

Company Registration: 3152737 - Place of registration: England

All the information contained within this electronic message from
Claranet Ltd is covered by the disclaimer at
http://www.claranet.co.uk/disclaimer

---

From: "iLya" <[email-masked]>
Date: Thu, 24 Mar 2011 17:05:10 +0100

> stupid is the way to build robust protocols (no buttons, no unneeded fluff

On this I'm with Claudio - big messages should be supported either by both 
sides or by none.

/iLya

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 24 Mar 2011 12:22:44 -0400

--
Jakob Heitz. 510-566-2901

> Why not? we do this with TCP already!

No you don't. TCP options size is limited to 40. God help you if anyone figures out how to increase it to 65535. The TCP data field is not interpreted by TCP, only the header. 

> iEYEARECAAYFAk2LQw8ACgkQtFWeqpgEZrL6xACbBjo8m0/Jm6X0lazpQinhL20c
> r1QAn1J8jk1QpiEzscLsz4oPGemBK05f
> =bcYt

---

From: Claudio Jeker <[email-masked]>
Date: Thu, 24 Mar 2011 17:42:19 +0100

> the min size you need to interop is 4096.

At the same time you need to interop on the new big messages.
 
> be important enough to require both peers to support these larger

Reading this mailing list in the last few days I came to the conclusion
that losing even a few bytes on notifications is a huge issue. So slamming
down the session with no reason at all will not be appreciated.

> negotiate this cap with 65535 then so be it.

It was suggested to support multiple sizes. Lets do a stupid little
example: lets guess I will implement a max-size of 23456 bytes (I like this
number) and someone else does something else like 16384 bytes for whatever
reason and then there is one system that supports 32000 bytes.
Now suppose you run this network and currently the biggest updates for a
single prefix is around 15000 bytes (huge amount of path attributes).
All is fine until that day when someone adds another 2000 bytes of
attributes to these pathes. Suddenly part of the network behaves funky
even though all systems support RFC XYZ Extended BGP Messages. All systems
conform to the RFC but the operator is still screwed just because some
people think they need a button for something that should be fixed number.
The maximum message size does not need to be negotiated. It is either 4096
or the extended size.
 
> with larger TCP MSS and BGP message sizes (for the UPDATE)

The time to transport the updates can be neglected. Most time is spent to
update the RIB (memory allocation, filtering, lookups).

For a good network stack the TCP MSS has no big influence on transfer
speed. It will just generate more packets. TCP window sizes are another
matter, though. Being able to add more NLRIs to a single update may help a
bit but I doubt it is really measurable (considering the current maximum
of 4096 byte messages). There are not many pathes that need to be sent
multiple times because the limit is hit.
 
> lost/corrupted) is another issue!

This is TCP's business and it works already for connections around the
world. I see no issue at all since the retransmit behaviour is not altered
by bigger BGP messages.

-- 
:wq Claudio