Subject: [Idr] draft-ymbk-bgp-extended-messages implications on TCP windows

From: Jeffrey Haas <[email-masked]>
Date: Sun, 17 Apr 2011 14:51:15 +0000

Caveat: I'm not a TCP protocol expert nor do I play one at my employer.
Like many people on this list, TCP looks like a socket to me and the
majority of the pain I live with under TCP gets represented at the socket
level.

Support for extended messages probably should be predicated on support for
TCP window scaling on the TCP session.  Without this, the maximum TCP window
size is 65k.  While it's my belief that most current router stacks support
this, the draft probably should note that requirement.  This is a somewhat
messy requirement since you either have to know that support is a priori
there or be willing to have your BGP peering session capable of dynamically
moving you from one peer group for 65k peers to one that isn't based on what
TCP tells you is negotiated.  Your implementation must also dynamically
advertise the capability based on the TCP negotiation as well.

What I'm a bit more concerned about isn't large window support as it is with
the interaction of larger message sizes and one's peers.  When an
implementation slows down its read behavior for whatever reason, this is
experienced as throttling at the TCP level and thus blocked writes at the
sender.  Implementations for efficiency sake often have hooks into TCP to
watch for a given low water mark for purposes of even trying to do a write
on the socket.

To continue on Jakob Heitz's issue about hold timers expiring on slow reads,
when one hooks into the TCP window state for purposes of determining whether
one can write or not, one now has to consider the need for larger windows.

We're unlikely to really require 65k packets.  However, we'll probably want
to change those marks for something bigger - perhaps 8-16k.  I already see
headaches in troubleshooting slow peers due to TCP stalling issues.  This
will only exacerbate some of them.

How is this going to impact your implementation?

-- Jeff

---

From: Jeff Wheeler <[email-masked]>
Date: Sun, 17 Apr 2011 15:03:43 -0400

> Like many people on this list, TCP looks like a socket to me and the

This is hard to explain.  From your perspective, the kernel TCP stack
provides a buffer which is NOT RELATED TO THE TX WINDOW SIZE.  This is
crucial to understand.

You can use setsockopt(2) or sysctl to tune the size of your kernel Tx
buffer on a per-socket or system-wide basis, as desired, to relieve
the user-space program ("rpd") of continually serving a user-space Tx
buffer.  In fact, it is a pretty good idea to raise the size of the
kernel buffer and reduce the size of the user-space buffer, because
the kernel can serve many sockets with pending Tx data in a single
context swap and without consideration for the amount of time rpd may
spend evaluating routing policy or doing other expensive operations,
since you have a preemptive multi-tasking OS.

The per-neighbor transmit buffer for BGP messages in "rpd" should be
large enough to hold:
* at least one message; and
* maybe as large as however many messages as may be generated towards
a single neighbor by one traversal of the event loop (which will
initiate a context swap anyway whenever it serves write-able
descriptors)
  + basically, as much memory as you want to trade for pre-computing
outgoing messages
    - because this pre-computation and buffering might be prioritized
in some way

> sender.  Implementations for efficiency sake often have hooks into TCP to

If you familiarize yourself with the original TCP specification,
you'll see that TCP Window is a flow-control mechanism.  The receiving
side has only so much kernel-space buffer (again, normally more than
the Window size) and it ACKs segments as they go into that buffer, but
scales down the Window size if user-space is not reading from the
socket often enough to keep enough free receive buffer to receive an
entire Window worth of data.  This is how TCP avoids spurious
retransmits because of the "kernel waiting on user-space" and it is
the reason for TCP window scaling.

On the transmitting side, the way you avoid the appearance of a
"stalled" session is by raising the value of SO_SNDBUF to buffer more
messages in the kernel, so they can be transmitted by the kernel as
soon as the receiving neighbor is ready for them, instead of waiting
for "rpd" to compute more outgoing updates.

On the receiving side, the value of SO_RCVBUF can be raised so its
kernel will buffer more incoming data, and send more ACKs, regardless
of whether or not "rpd" has actually read them out of kernel-space
yet.  This allows the TCP Window to scale up as-needed and reduces the
appearance of "stalls" on the transmit side, provided that the
transmit side is regularly writing new messages into its kernel
buffer.

> watch for a given low water mark for purposes of even trying to do a write

This adds implementation complexity to user-space, or "rpd."  The
benefit is that you will have less expensive executions of the code
that serves write-able descriptors, but the cost is you will execute
some code anyway only to decide not to write anything at all from the
user-space buffer to the kernel-space buffer.

To be honest, it is not an optimization -- it is stupid.  The reason
why is those less-expensive executions of the "socket is write-able"
code cause context swaps that don't actually do anything at all, where
you could have fewer context swaps, which are slightly more expensive
(write to socket and then manage user-space buffer), but cause the
kernel-space buffer to become full again, preventing any more context
swaps until some in-flight TCP segments have been ACKed and more
kernel-space buffer is available.

If your developers are really peeking into the TCP stack to find out
how much more send buffer is available, they should learn a bit more
about the underlying TCP stack in your OS and take advantage of its
features.

If the concern is that JUNOS is moving toward prioritizing route
update computations and message transmissions, they should structure
the user-space buffers as follows:
* pointer to message fragment currently waiting to be sent (really a
whole message but pointer may be mid-message)
* list of high-priority messages pending transmission, some of which
are already pre-computed
* knowledge of whether high-priority route updates are pending computation
* list of lower-priority messages pending transmission

If they are concerned about a large number of low-priority messages
being stored into the kernel-space buffer, thus the TCP stack will
send them before new high-priority messages, they must realize that
there is an intrinsic trade-off between a larger kernel-space buffer
and lower-latency transmission of high-priority messages and buffer
management complexity.

In either case, though, peeking into the TCP stack is pretty foolish
because it dials up the number of context swaps, and much more benefit
can be found in terms of prioritizing updates in terms of the "rpd"
"work list" than in tinkering around with the buffer manager and TCP
stack.  After all, the updates themselves are generated by rpd as it
"does work" in an order which must be prioritized to see any practical
gain.

> one can write or not, one now has to consider the need for larger windows.

No, you really do not need to consider TCP Window value at all.  There
are several kinds of BGP message type that do not satisfy the hold
timer's need to receive a message before closing the session -- a
buffer full of large NOTIFICATION messages would not do so.  The
specification should be altered as Jakob suggests to satisfy the hold
timer upon receipt of either any message-type (IMO) or any data
(Jakob's suggestion) for slow TCP sessions.

I am not concerned about a session reset if 64KB has not been received
yet, but I do understand why it is concerning if a bunch of
NOTIFICATION messages have been buffered and are pending transmission,
and no UPDATE or KEEPALIVE messages are being sent for a long period
of time towards a slow neighbor.

To put this in context, though, even a 64Kbit/sec leased-line can
still receive a 64KB message in well under the default hold-time for
BGP, as long as it does not need to receive several large NOTIFICATION
messages in sequence.

Either requiring receipt of any message type, or any data, would be
satisfactory and avoid this problem.

> will only exacerbate some of them.

This is understandable if you are trying to "optimize" transmission by
peeking into the TCP stack, and blowing up the number of CPU context
swaps, instead of simply letting the TCP stack do its job -- which is
to allow rpd to spend more time "working" and less time dealing with
transmit buffers.

Any TCP stalling issues you are encountering are a product of your own
making.  If you don't take advantage of the kernel-space transmit
buffer, then the neighbor won't get more messages until rpd finishes
"whatever it is working on" and returns to the event loop, where
write-able descriptors can be serviced.

> Like many people on this list, TCP looks like a socket to me and the

To come back to the first lines in your original message, if TCP looks
like a socket to you, and you have stalling problems, all you need to
do is grow the kernel transmit buffer (realizing that there is a
trade-off if you try to prioritize messages) or return to the event
loop more often (harder.)  Really, in an ideal implementation, "work"
evaluating policy would not block pending writes from the prioritized
message queue to the kernel TCP buffer; but this really requires
either more code complexity in a single-threaded program, or a
thread/process dedicated to the queue/buffer manager.

But if you don't care about prioritizing updates toward the
kernel-level TCP buffer, this is a total non-issue.  Anyone having TCP
stalling *without* trying to prioritize update messages is just
reaping what they sowed by trying to "optimize" something that was
already optimal -- the kernel efficiently handling I/O with fewer
context swaps.

-- 
Jeff S Wheeler <[email-masked]>
Sr Network Operator  /  Innovative Network Concepts

---

From: Jeffrey Haas <[email-masked]>
Date: Sun, 17 Apr 2011 20:15:10 +0000

> provides a buffer which is NOT RELATED TO THE TX WINDOW SIZE.  This is

I don't believe you just wrote this list TCP-101.

For your clarification, I'm not an IETF TCP subject matter expert.  I.e. I
don't contribute to tcpm.  I don't maintain our tcp stack.  I'm well aware
of how stream sockets work under the covers and what things look like on the
wire, thankyouverymuch.

Secondarily, this is not about Junos nor rpd.  It's generically about TCP
and BGP and the fact that BGP implementors get to see a lot of the transport
level stupidity reflected at the socket level.  And I see this stuff from
the tcpdump analysis level as I get to prove out why some other
implementation is stalling out for no good reason and why it only hands back
window sizes in certain levels of chunking.  And why in our implementation
we have to take account of the to-and-fro of windowing based on that
behavior.

I'm going to trim a massive portion of your rpd speculation since some of
it's right and some of it's very wrong. Mostly it's also largely irrelevant to
my original point.

> > watch for a given low water mark for purposes of even trying to do a write
[...]
> can be found in terms of prioritizing updates in terms 

Not true.  Unlike a web server or less PDU-oriented protocols, BGP speakers
need to spend as much time making sure that there are entire BGP messages
available for the peer.  Partial messages in the kernel lead to partial
messages on the wire which, if not drained in a timely fashion, stall the
remote peer.

It makes no sense to bother the routing process for a packet to transmit if
there's either no packet to transmit or not enough room to transmit one.

The dynamics of "enough room" are what will be changing with the extended
message feature.

> satisfactory and avoid this problem.

You apparently haven't noticed that's not how 4271 is written.

That said, the 4271 rev of the BGP FSM misses a lot of reasonable details
for an implementation.  

> > We're unlikely to really require 65k packets. ?However, we'll probably want
> > to change those marks for something bigger - perhaps 8-16k. ?I already see
> > headaches in troubleshooting slow peers due to TCP stalling issues. ?This

In any routing protocol implementation, if you have state you need to
transmit, you want to get it to the remote side as quickly as possible with
as little work as possible.  The conditions of concern are when the kernel
TCP buffers are mostly full.  One way or the other, you will have to put
your new work into them.  The message sizes impact how often and how
efficiently you can do the work to fill that buffer.

In other words, message size impacts work done and its timeliness.  This is
the whole point of the thread.

> making.

If that were only true.  I can fix my bugs, I can't fix other people's.
I just have to deal with them.

-- Jeff

---

From: Jeff Wheeler <[email-masked]>
Date: Sun, 17 Apr 2011 18:12:42 -0400

> I don't believe you just wrote this list TCP-101.

I have been in equal disbelief each time I've seen a reference to TCP
Window size in this context.  It is abundantly clear that some posters
do not know how TCP works.

> level stupidity reflected at the socket level.  And I see this stuff from

No, any "transport level stupidity" you see is not at the socket
level.  Again, if you are peeking into the TCP stack to find out how
much more kernel-space buffer is available, instead of simply filling
it up, you are adding context swaps and complexity.  Unless your
intent is to prioritize outbound messages, there is zero benefit to
this.

> we have to take account of the to-and-fro of windowing based on that

No, you do not need to account for "some other implementation"
stalling -- you only need to have data in the kernel buffer for
transmission when the neighbor is ready to receive said data.  Keeping
a large user-space transmit buffer creates Tx stalls.  Having a
too-small kernel-space receive buffer gives the appearance of "Rx
stalls," but in fact, the TCP Window is a flow-control mechanism and
it is not incorrect to take advantage of it.

> it's right and some of it's very wrong. Mostly it's also largely irrelevant to

What is irrelevant is your original post.  TCP Window size has no
bearing on maximum message size.  My "rpd speculation" is for your
benefit -- if you need to understand how prioritizing message
transmission and avoiding stalls can be conflicting goals, it's in
your inbox.  This is the only reason you would have to poke around at
the TCP stack from user-space.

If this is not among your goals, then fooling with the TCP stack, and
having an overly-large user-space buffer without understanding the
function of the kernel-space buffer, is stupid.

If you care to clarify how abstracting the function of rpd into "work"
and "queue/buffer manager" to explain the program's interaction with
the TCP stack, and remote neighbors, is "very wrong", please feel free
to do so.  Otherwise, I'm happy to write that off as defensive
grand-standing, as it should be abundantly clear to anyone who does
understand TCP that you are complaining about self-made stalling
problems.

My use of rpd as an example is also to make this explanation distinct
from BGP speakers which choose to implement an independent TCP stack
in the BGP speaker, such as IOS.  In the case of IOS, the BGP Router
process is already executing anyway, so there is not an additional
performance penalty for the BGP speaker having intimate awareness of
in-flight segments vs buffered data.  There obviously is a complexity
penalty, as we can see when troubleshooting things that look like "BGP
bugs" which sometimes turn out to be "IOS BGP-speaker TCP bugs."

> messages on the wire which, if not drained in a timely fashion, stall the

The possibility of partial messages on the wire always exists.  You
can't get rid of partials without doing all of:
* make BGP message smaller than MSS
* turn off Nagle
* turn off retransmit coalescing
or use a different transport protocol than TCP.

The next best way to avoid partial messages, which is just as
effective as peeking into the TCP stack, is to use SO_SNDLOWAT.  This
does exactly what you describe, "ensure there is enough room to
transmit an entire packet" into the kernel-space buffer, still without
requiring the programmer to be concerned about the TCP Window.

Obviously, the send buffer must be large enough to store one whole
message, but again, this buffer is unrelated to the TCP Window size.
It is a convenience to allow the user-space program to spend less time
servicing a message buffer/queue and write-able descriptor, and more
time "working."

> there's either no packet to transmit or not enough room to transmit one.

See above for how this is accomplished without fooling with the TCP
stack.  In either case, it is up to you whether or not you decide to
buffer partial-messages in the kernel or not.  They absolutely will go
"on the wire" as partials no matter what you do.

> the whole point of the thread.

Yes, but none of this is related in any way to TCP Window size.

> I just have to deal with them.

I do sympathize with your desire to only place whole-messages into the
TCP stack Tx buffer.  Unfortunately, since BGP messages can be larger
than TCP MSS, there is no guarantee that partial messages won't be
transmitted.  You are lumping some unrelated issues into "TCP Window,"
which really has nothing to do with this.