Subject: [BEEPwg] rfc 3081, queueing, implementations

From: james woodyatt <[email-masked]>
Date: Mon, 11 Mar 2002 17:48:41 -0800

fellow travelers--

Consider that RFC 3081 has this to say in Section 3.1.4, Use of Flow 
Control:

> Implementations may wish to provide queue management facilities to the 
> application using BEEP, e.g., channel priorities, (relative) buffer 
> allocations, and so on.  In particular, implementations should not 
> allow a given channel to monopolize the underlying transport window 
> (e.g., slow readers should get small windows).

I am completely unfamiliar with what other implementations do in this 
regard (since I still have a long way to go before I begin 
interoperability testing), but I'm interested to know what facilities 
are in use now.

One thing that concerns me: the proposed standard does not define a way 
for a BEEP application to signal the desired channel priority or 
relative buffer allocation to its peer.  I'm not sure such facilities 
are very useful without such a mechanism.

Another thing that has me worried: I'd like to understand how giving 
smaller windows to slower readers prevents a given channel from 
monopolizing the underlying TCP window.  Could someone point me to an 
explanation of the theory behind this approach?

--
j h woodyatt <[email-masked]>

_______________________________________________
BEEPwg mailing list
[email-masked]
http://lists.beepcore.org/mailman/listinfo/beepwg

---

From: Gabe Wachob <[email-masked]>
Date: Mon, 11 Mar 2002 18:27:56 -0800 (PST)

> interoperability testing), but I'm interested to know what facilities

PyBEEP has (when completed) a pluggable "scheduler". This scheduler could
be configured to give channels different "priorities" based on almost any
algorithm you could choose to implement. Initially, there will only be a
round robin scheduler implemented, most likely. (That is, roughly,
rotating through the list of "ready to write" channels to write from).
This, of course, only directly affects "priority" only as far as *writing*
data to remote peers.

> are very useful without such a mechanism.

This is a "topside" (or API) specification and is not addressed by BEEP,
just as HTTP doesn't address such things as CGI. Perhaps all the
implementations will converge on some API or at least concepts, but I
think its a little too early quite yet, and even so, it wouldn't be
generally applicable for all BEEP-based applications.

I saw a draft by someone (which is no longer current) suggesting some ways
of negotiating "priorities" between peers (which is a separate issue) -
I'm hoping someone will speak up as to the document title and URL.

> explanation of the theory behind this approach?

Thats hard to do in a short email, and I'll probably butcher it, but I
think the idea is that a slow reader is going to leave data in the
incoming TCP stream buffer longer (filling the TCP window) and thus
doesn't allow other fasters readers (who would empty incoming TCP stream
buffers quicker) to use the connection while the slow reader is consuming
the data.

I think also that windowing manipulation allows for a local peer to
influence the ability of a remote peer to send data to "higher priority"
local profiles/channels (even if the local profile/channel is not a "slow"
reader).

This stuff is easy to make misstatements about, so I'll stop here ;-)

	-Gabe

-- 
Gabe Wachob                   [email-masked]
Personal                   http://www.wachob.com
CTO, WiredObjects    http://www.wiredobjects.com

---

From: james woodyatt <[email-masked]>
Date: Mon, 11 Mar 2002 19:10:32 -0800

On Monday, March 11, 2002, at 06:27 PM, Gabe Wachob wrote:
>> [...]
> algorithm you could choose to implement. [...]

That's good to know.

> generally applicable for all BEEP-based applications.

My concern is with the subclass of BEEP applications for which some kind 
of queue management facility *is* required.  Without a standard 
mechanism for signaling queueing parameters to a BEEP peer, all such 
applications will inter-operate only with peers built around the same 
queue management facility.  Seems like an opportunity for vendors to 
"embrace and extend" the BEEP protocol at the expense of the general 
Internet welfare.

At least, that's the way it seems to me right now.  I hope I'm wrong 
about that.  If I'm right, then I humbly suggest that an appropriate 
amendment should be considered before the protocol advances to Draft 
Standard.

> buffers quicker) to use the connection while the slow reader is 

This doesn't make as much sense to me as I would like.  I think that RFC 
3081  encourages implementations to read segments out of the TCP stream 
as soon as they arrive, pushing the frames they contain into per-channel 
hold-off queues.  That's how I'm writing my implementation.

Here's where my concern lies: if a channel at your peer is quickly 
sending really small messages and slowly receiving really large replies, 
then my peer giving that channel a smaller window might seem like "a 
good idea" intuitively, but I don't understand how it actually helps 
prevent that channel from "monopolizing the underlying TCP window" in 
any useful way.

Is this believed to be an application-specific issue?  If so, then I 
suppose I need to adjust my thinking about how cleanly the BEEP protocol 
separates the application layer from the transport layer.

Is there even an issue with regard to channels "monopolizing the 
underlying TCP window" at all?  I find this phrase in RFC 3081 fairly 
confusing, actually.

> This stuff is easy to make misstatements about, so I'll stop here ;-)

Anybody else want to help straighten me out?

---

From: Gabe Wachob <[email-masked]>
Date: Mon, 11 Mar 2002 19:35:58 -0800 (PST)

> "embrace and extend" the BEEP protocol at the expense of the general

I hear this, but you could just define your profile with this behavior and
then require that multiple channels be controlled by the same object which
can orchestrate the 'window sizes' among the channels it controls (which
all the channels for a given underlying BEEP connection).

> amendment should be considered before the protocol advances to Draft

Well, BEEP is cooked and delivered RFC 3080/3081 means its not going to
change for a while.

> hold-off queues.  That's how I'm writing my implementation.

Yes, I'm putting into hold-off queues, but those queus could fill up
quite quickly. Essentially our implementations have very large buffers,
and as such the windowing stuff seems less important. I think the key is
this (RFC 3081, sec 3.1.4):

"each time a frame is received, a SEQ frame should be sent whenever the
window size that will be sent is at least one half of the buffer space
available to this channel; and"

The idea is that if a receiving application is about to fill its
*application-layer*/channel buffer, then this will back up into TCP-layer
buffers and the problems I mentioned will ensue.

You should probably check out RFC1122 which I should check out again as
well.

> prevent that channel from "monopolizing the underlying TCP window" in

Because the data is not being sent to the receiving side at all until the
channel profile object has a fighting chance to process it. In the
busy-reader state, the BEEP connection may in fact be quiescent even if
there is only one channel on the reading side.

If another channel started up then, and its window stayed essentially open
all the time, it would get full use of the BEEP channel instead of waiting
for the slow-reading channel to read data out of the TCP buffer. (and when
the slow reader did get open space in its window, it would presumably only
get the data it could *immediately* process or buffer at an *application*
layer).

Does that make sense? Again, its hard to have this discussion
non-graphically ;-)

> separates the application layer from the transport layer.

I'm not sure what you mean by application-specific here, so I'm not sure I
could give you a good response.

> underlying TCP window" at all?  I find this phrase in RFC 3081 fairly

I feel the need for some visio diagrams. I'll try to throw together some
visio diagrams tonight and post a URL to the list to describe what I
believe to be the situation trying to be avoided and the way in which it
is avoided with the flow control stuff in BEEP.

	-Gabe

---

From: Marshall Rose <[email-masked]>
Date: Mon, 11 Mar 2002 20:47:12 -0800

hi. i think that the basic problem is that you are reading too many things into the tcp mapping specification. the notion of priorities, if you choose to implement it, is a local mechanism, not an end-to-end mechanism. hence, there is no need for any kind of negotiation mechanism.

in the beepcore-tcl implementation, for example, the application can tell the beep layer the "relative weight" of the incoming bandwidth that should be reserved for each channel. the implementation itself uses this in conjunction with a round-robin algorithm in order to determine what goes next and how much (with channel zero always getting taken care of first). one could generalize this approach by adding a priority in addition to the relative weight. if your familiar with the way the SRV algorithm works, you'd understand the generalization. this is probably overkill, but it may get implemented anyway.

other implementations are free to provide whatever local mechanism that's convenient, useful, etc. the only requiremen that the tcp mapping specification makes is that you have to be able to convey to the other side the window size. how you choose the window size is up to the implementation and the application making use of it.

/mtr

---

From: Allison Mankin <[email-masked]>
Date: Tue, 12 Mar 2002 00:53:08 -0500

Hi,

A missing point in the discussion has been that the receiver doles out
the window to the sender for each channel, using SEQ Frames for 
each of the sender's channels.

Consider a path from Sender (S) to Receiver (R) of 100 Mbps.  Consider
a simple example of three unidirectional (that is, S is only a data
sender and R only a data receiver) channels, C1, C2 and C3.  Assume
that S always has data ready to send on each of the channels (for the
many applications of BEEP that are not like this, TCP/BEEP flow control
issues don't cause any concern).

If R wants C1 to have 20 Mbps, it opens the window for S in C1 using
SEQ Frames and advancing the ackno/window the 20 Mbits each second.
S must stop sending data in channel C1 when C1's window is full and it
can't send into C1 again till the next SEQ Frame for C1 is received from R.
In concept, this means C1 will average 20 Mbits per second over time.
(Meanwhile R offers C2 and C3 windows using SEQ Frames for them).

This is a most simplified example.  Windows don't map neatly to rates
(equations such as those described in draft-ietf-tsvwg-tfrc-03.txt are
needed).  The example also oversimplifies by assuming the available
capacity from S to R stays at 100 Mbps in a real network path, whereas
available capacity for the underlying TCP connection is dynamic, as TCP 
adapts to congestion and to other flows sharing resources with it.  The windows
for C1, C2 and C3 really need to add up to the TCP Congestion Window (CWND),
the lowered window size that the TCP level sender obeys, based on the TCP
connection's estimates about other flows and congestion.

When CWND is lower than the sum of R's BEEP-level windows for C1, C2
and C3, the channels may get unfair amounts of capacity.  The TCP 
connection might have a real 10 Mbps due to congestion. R offers C1 20 Mbps,
and this leaves C2 and C3 unserved for some time period.  If S round-robins
through its channels, they can avoid starvation, but performance could
be unfair or unpredictable.  If S and R are too clever with window 
tuning and priorities, the results could be dismal.  Another way to
state Marshall's mail on this thread a little earlier is to recommend
simplicity in the use of the BEEP TCP mapping's flow control features.

>    In particular, implementations should
>    not allow a given channel to monopolize the underlying transport
>    window (e.g., slow readers should get small windows).

I think this should say "slow readers should offer small windows".  
IESG review missed this point before :)

> 
>    In addition, where possible, implementations should support transport
>    layer APIs that convey congestion information.  These APIs allow an
>    implementation to determine its share of the available bandwidth, and
>    also be notified of changes in the estimated path bandwidth.  Note
>    that when a BEEP session has multiple channels that are
>    simultaneously exchanging large messages, implementations without
>    access to this information may have uncertain fairness and progress
>    properties during times of network congestion.

The above paragraph is (I hope) explained more by this mail.  An API such
as it recommends is found in RFC 3124 (The Congestion Manager), which came 
out after RFC 3081 was ready to be approved.    Using the CM API, S could
query its TCP socket to make decisions on when to stop sending into 
each channel based on CWND directly (perhaps weighted proportionally to the 
channels' windows given it by R).

I'd be interested to know if this approach could be taken by the 
implementations that are being developed. 

Allison

---

From: Marshall Rose <[email-masked]>
Date: Mon, 11 Mar 2002 22:10:06 -0800

> IESG review missed this point before :)

right you are.

> implementations that are being developed. 

is there a url with implementation pointers for the CM? if so...

/mtr

---

From: Allison Mankin <[email-masked]>
Date: Tue, 12 Mar 2002 01:17:32 -0500

> is there a url with implementation pointers for the CM? if so...

http://nms.lcs.mit.edu/software/CM/ (implementation)

http://nms.lcs.mit.edu/projects/cm/ (papers etc).

---

From: Gabe Wachob <[email-masked]>
Date: Mon, 11 Mar 2002 23:00:16 -0800 (PST)

Allison-
	Very interesting stuff, of course.

	I put together some documentation exploring (for personal
understanding as well as to show to others) using "immediately
emptied" buffers  where the data is consumed immediately upon receipt
into a channel. Thus, SEQ messages always end up sending SEQ frames with
window sizes equal to the entire buffer size for the channel.

http://beepnik.wiredobjects.com/bandwidth/

	The gist of the demo (pictures) is that effectively the relatively
size of the buffer a channel has determines the relative bandwidth that a
channel gets. Thats well understood (but not well documented with
easy-to-understand examples w/barney-like even-the-kids-get-it pictures).

	So thats my baseline - I'm making some assumptions (esp. about
constant bandwidth) that you point out are not realistic, but I'm
wondering how much that modfies my results..

	I'm looking for feedback on the document linked above.

	More comments inline:

On Tue, 12 Mar 2002, Allison Mankin wrote:

<snip>

> issues don't cause any concern).

I believe my document makes the same assumptions (at least for most of the
time steps in my demo).

> SEQ Frames and advancing the ackno/window the 20 Mbits each second.

Well, it would only "advance" the window when it received data, so it
would only advance the window at most at the rate it received data, no?
In other words, wouldn't advancing the window 20mbit/sec would only happen
if data was coming in at 20mbit sec?

<snip>

> The windows

Right, because then you are going to run into TCP congestion, in which
case managing the channel windows gets less useful because the
relationship between elapsed time and size of the window is thrown off
(not constant) and buffer size doesn't map directly to bandwidth... Or am
I completely off?

> and this leaves C2 and C3 unserved for some time period.

But R isn't offering bandwidth, its offering buffer space (window), isn't
it? This is where I start to get lost.

<snip>

I think I understand why you want to avoid triggering TCP congestion
protocols, but I'm not following the part about unfairness between
channels.

	-Gabe

---

From: "Randolph S. Kahle" <[email-masked]>
Date: 12 Mar 2002 04:05:31 -0700

I think I am getting confused about the purpose of the BEEP protocol.

Maybe someone can re-ground me.

It seems that the discussion in this thread is focused on adding
complexity to a protocol that creates channels of communication on top
of a single TCP/IP connection.

What is the point of this complexity?

Why not keep BEEP as simple and clean as possible, open up multiple
TCP/IP connections and allow the work that is being done on IP routing
and the TCP/IP protocol handle these complexities?

What application level problems does this complexity address or solve?

-- Randy

---

From: Allison Mankin <[email-masked]>
Date: Tue, 12 Mar 2002 08:57:41 -0500

Randy,

The BEEP TCP mapping already has the complexity - it comes with
the territory - my mail and Gabe's did not make it up.
My point was to explain the situation when flow control has to
come into play and advise that only dead simple use of the flow control
feature be made.  And as I said, I expect a lot of BEEP usages aren't
network-limited (don't have data ready to send all the time), in which 
case they don't have to think about flow control at all.

> and the TCP/IP protocol handle these complexities?

There was work on this, but it would have to avoid the performance
negatives of those TCPs competing with each other and overcompeting
along the path, and many other problems that HTTP found with multiple
parallel TCPs.  TLS use would be far less efficient as well.

> What application level problems does this complexity address or solve?

It addresses the fact that BEEP is trying to make a threaded
service on top of an unthreaded service (TCP) to get a lot of other
application architecture benefits.  Sometimes there's a cost.
_______________________________________________
BEEPwg mailing list
[email-masked]
http://lists.beepcore.org/mailman/listinfo/beepwg

---

From: Marshall Rose <[email-masked]>
Date: Tue, 12 Mar 2002 08:48:30 -0800

> application architecture benefits.  Sometimes there's a cost.

i agree!

of course, what often goes unsaid (when this thread shows up twice a year), is that if you don't want the benefit, there really isn't a cost. if you want to implement a BEEP layer that doesn't do multiple channels, then you can hardcode all of this stuff quite simply...

/mtr

---

From: Darren New <[email-masked]>
Date: Tue, 12 Mar 2002 09:50:44 -0800

And, for what it's worth, the beepcore-c implementation sends a SEQ and
a frame each time the "core" library gets invoked. (Of course, it
doesn't send SEQs if there's no windows needing opening, and it doesn't
send frames if there's no frames with window to send them in.) The SEQ
it sends is the one that would open the window the most. I.e., it sends
the SEQ for the channel with the most available buffer space. The frame
it sends is the frame on the highest-priority channel that can be sent
right now, accounting for tuning resets, ordering of outgoing messages,
and available window size. I'm not sure if the core-level priorities got
reflected in the higher-level APIs (it's just a number you can change),
but if not, the channels opened later get priority over those opened
earlier, IIRC.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
  To the user, everything works just as expected,
    assuming the user's expectations are correct.
_______________________________________________
BEEPwg mailing list
[email-masked]
http://lists.beepcore.org/mailman/listinfo/beepwg

---

From: james woodyatt <[email-masked]>
Date: Tue, 12 Mar 2002 12:16:54 -0800

On Tuesday, March 12, 2002, at 08:48 AM, Marshall Rose wrote:
> of course, what often goes unsaid (when this thread shows up twice a 
> year), is that if you don't want the benefit, there really isn't a 
> cost. if you want to implement a BEEP layer that doesn't do multiple 
> channels, then you can hardcode all of this stuff quite simply...

Agreed.  And I'm not complaining about the cost.  Quite the opposite.

I, for one, am keenly interested in this feature of the BEEP protocol, 
which is why I am being an annoying git and trying to make sure I adhere 
strictly to the letter and spirit of the protocol specification.

I'll have a reaction to Allison Mankin's post later this evening when I 
have some free time again to write it.

---

From: james woodyatt <[email-masked]>
Date: Tue, 12 Mar 2002 21:04:26 -0800

> each of the sender's channels.

I had assumed we were all familiar with the specification.

> issues don't cause any concern).

Your example is a generalization of the case that has me concerned.

> (Meanwhile R offers C2 and C3 windows using SEQ Frames for them).

What if the sender (S) is where the application needs to make decisions 
about how much of the transport window each channel (C1, C2 and C3) 
should get relative to one another?  How are those parameters 
communicated to the receiver (R) where they are to be used?  What *are* 
the parameters?

Obviously, the BEEP protocol (specifically the TCP mapping) leaves this 
to the application layer to decide.  Hence, my concern-- stated in a 
previous message-- that without a standard facility to manage flow 
control, BEEP applications of this subclass will only be able to 
inter-operate effectively with peers that are constructed using 
implementations that sport compatible congestion management facilities.

I gather from Marshall's post that this is not widely regarded as a 
problem.  I can easily live with that.  It helps to have a clarification 
of the intent of the working group.

> needed).  [...explanation of congestion management issues elided...]

> IESG review missed this point before :)

That makes much more sense.  I'd go a little further:

	In particular, implementations should permit applications to
	prevent a given channel from monopolizing the underlying transport
	window (e.g. slow readers should be able to offer small windows).

I think this makes it clear that applications-- not the framework-- are 
responsible for balancing channel flows in the face of transport 
congestion.

In fact, now that I think about it: every transport mapping will have 
some kind of "underlying transport window" for which congestion 
management may be a concern.  (For example, I'll encounter the same 
basic issue should I go completely insane and try to implement a 
transport mapping for POSIX shared memory and semaphores.)

Unfortunately [or not, depending on your viewpoint, I guess], there is 
no facility in the BEEP core itself to balance channel flows in the face 
of transport congestion, and the TCP mapping document leaves the matter 
entirely open to the implementation of the receiving peer.

That doesn't seem to bother anyone else, so I guess I should shut up 
about it-- and be happy the specification is 1) a Proposed Standard, and 
2) isn't going through any serious revision thrash at the moment.

--
j h woodyatt <[email-masked]>
"You're standing on sacred ground.  Many strange and wonderful
things have transpired right where you're standing."  --unattributable

---

From: "Randolph S. Kahle" <[email-masked]>
Date: 13 Mar 2002 03:53:26 -0700

> of course, what often goes unsaid (when this thread shows up twice a year), is that if you don't want the benefit, there really isn't a cost. if you want to implement a BEEP layer that doesn't do m…

Now I am even more confused than before.

Are you saying that an implementation of the BEEP protocol can support
only one channel and still be considered compliant?

-- Randy

---

From: Marshall Rose <[email-masked]>
Date: Wed, 13 Mar 2002 07:29:10 -0800

> only one channel and still be considered compliant?

a minimal, non-useful beep implementation does only channel 0. nowhere in the specs does it say that you have to let the other side start a channel.

a special-purpose beep implementation (e.g., one built for doing only sacred or reliable syslog) does channel 0 and one channel for the "real work". because tuning channels are used serially, the implementation might also support tls and a couple of sasl profiles.

as long as the implementations generate valid responses to the start requests, there's nothing out of compliance.

the three beepcore implementations are intended for general-purpose use, so they don't follow this philosophy. as a result, they have a lot of mechanisms for supporting the details of multiple channels operating in parallel.

a special-purpose implementation can hardcode the vast majority of this stuff.

/mtr