Subject: [BEEPwg] Channel response serialization

From: "Paul Andrews" <[email-masked]>
Date: Wed, 24 Jul 2002 16:37:07 -0400

I'm curious about the justification for the requirement that responses to
queries on a channel be in the same order in which the queries were issued.
I understand that it has been stated that this simplifies the protocol
implementation. Does this mean that it simplifies the protocol or the
libraries that implement it? If its just the libraries, couldn't the
serialization requirement be made optional (i.e. part of the session
negotiation)?

I ask because it seems that this excludes a large class of applications from
using BEEP, so that those of us who need truly asynchronous messaging are
left with the same problem that BEEP was intended to solve. Namely: HTTP
doesn't quite fit and SMTP doesn't quite fit.

Now BEEP doesn't quite fit either :-(

---

From: Jered Floyd <[email-masked]>
Date: 24 Jul 2002 18:50:56 -0400

> I ask because it seems that this excludes a large class of
> applications from using BEEP, so that those of us who need truly
> asynchronous messaging are left with the same problem that BEEP was
> intended to solve. Namely: HTTP doesn't quite fit and SMTP doesn't
> quite fit.

In what situations is the opening of multiple channels insufficient
for your purposes?

--Jered

_______________________________________________
BEEPwg mailing list
[email-masked]
http://lists.beepcore.org/mailman/listinfo/beepwg

---

From: Marshall Rose <[email-masked]>
Date: Wed, 24 Jul 2002 15:50:37 -0700

> doesn't quite fit and SMTP doesn't quite fit.

hi. you're right: it's probably a silly limitation in that the only thing that it makes "simpler" is the implementation on the side receiving the responses. (the side generating the responses couldn't care less, right?)

i guess my question though is why not use the MSG/ANS approach to dealing with this? without understanding what the traffic looks like for the class of problems you're interested in, i can't really comment more.

/mtr

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 09:18:10 -0400

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Marshall Rose
> Sent: Wednesday, July 24, 2002 6:51 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [BEEPwg] Channel response serialization
>
>
> applications from
> hi. you're right: it's probably a silly limitation in that the
> only thing that it makes "simpler" is the implementation on the
> side receiving the responses. (the side generating the responses
> couldn't care less, right?)
>
> i guess my question though is why not use the MSG/ANS approach to
> dealing with this? without understanding what the traffic looks
> like for the class of problems you're interested in, i can't
> really comment more.

The application is one where many (i.e. hundreds or thousands) of
independent entities on one peer wish to send a request to a remote peer
acting as a server. The server process the requests in parallel and sends
responses to the requests as soon as it can so that senders don't block on
the slowest response. I think this answers your question about MSG/ANS. My
reading of the RFC implies that although multiple ANS can be sent in any
order to a single MSG. The processing of multiple MSGs on one channel is
still performed serially, right?

Now I could use a pool of channels. But this seems wrong for several
reasons:

1. I would like to use channels for semantic purposes - i.e. they are
strongly associated with the meaning of the message. If I don't do that then
I have to layer another addressing mechanism on top of that provided by
BEEP. BEEP, itself, associates channels with message purpose by assigning
channel 0 as the control channel.

2. If I'm going to have a pool of channels, why not just have a pool of
sockets instead? OK, I know there are some reasons but the more that I have
to layer on top of BEEP, the less compelling is the argument for using it.

3. If I can implement parallel MSG/RPY over a single logical channel, then
BEEP could do it over a single BEEP channel and save me the bother.

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 09:26:48 -0400

> [mailto:[email-masked]]On Behalf Of Jered Floyd
> To: Paul Andrews
> Cc: [email-masked]

It isn't insufficient in itself, but it prevents me from attaching a
semantic purpose to the channel number (all of the requests have the same
semantic here, just a different payload). It also means that I have to
implement yet-another-pool-mechanism to manage the allocation of channels in
a manner invisible to the application.

It struck me that that wasn't really the purpose of channels. I got the
impression that channels were intended to have a semantic purpose - and
that's how I would like to use them, e.g. channel 0 is a control channel -
you wouldn't allocate multiple control channels arbitrarily to get around
the serialization restriction otherwise how would BEEP know that they were
control messages?

Finally the restriction on reply ordering seems to be arbitrary and forces
people like me to put in a lot of extra work that we wouldn't have to do if
the restriction wasn't there (I know, we all want something for nothing
:-) ).

---

From: Jered Floyd <[email-masked]>
Date: 25 Jul 2002 10:01:25 -0400

> the serialization restriction otherwise how would BEEP know that they were

Hm.  I defer to Marshall on any statements regarding intent, but I
think that semantic binding is only one of the purposes of channels.
While the binding of some specific profile to each channel provides
semantic separation between channels, the serialization of messages on
a single channel allows the client to pipeline requests with a
guaranteed execution ordering.

In fact, this is how we make use of multiple channels when performing
complex tasks in our protocols.  We manage a pool of channels bound to
a given profile, and then assign messages to 'free' channels
(i.e. ones with no currently outstanding message), while honouring
data dependencies by pipelining requests on a single channel.

While I don't fundamentally object to the idea of allowing
out-of-order replies on a channel, I don't want to lose the guarantee
of in-order processing of subsequent messages on a single channel.  I
posit that in any protocol that allows for volatile data, there are
few situations in which one is likely to have in-order processing with
out-of-order replies (while maintaining correctness).

Question: If we remove the restriction on reply-ordering, do we then
also allow frames on multiple RPYs or ERRs to be interleaved?  That
seems like the logical next step.  Of course, then for parallelism we
really ought to allow frames of MSGs to be interleaved.  But, then
we're back to something that looks like the channel mechanism, though
less explicitly managed!

> the restriction wasn't there (I know, we all want something for nothing

PermaBEEP goes a step further towards enforcing this by not allowing
the application to retrieve the next message on a channel until it has
initiated a reply to the last message delivered to the application.
(Note that is initiated, not fully sent.)  This decision was a bit
contentious, but I think it best serves developers in that it prevents
them from shooting themselves in the foot in a system that dispatches
requests into a server framework.

If you think it would be interesting to explore, I believe it would be
relatively easy to modify PermaBEEP to remove both this and the
restriction on reply ordering.  I'm not sure there's a lot of value in
it, though.  (Hm, and I really must roll the 0.9 release soon; I meant
to do that last month.)

--Jered

---

From: Jered Floyd <[email-masked]>
Date: 25 Jul 2002 10:20:56 -0400

I'm not adding terribly much here from the message I just sent, but
I'll add my $0.02.

> independent entities on one peer wish to send a request to a remote
> peer acting as a server.

I believe that a (perhaps dynamically managed) pool of channels is the
right solution here.

> strongly associated with the meaning of the message.

I'm not sure I follow what you mean by this.  Is it that your profile
is not stateless, and so each channel has some 'defaults' that make
multiple channels operating with the same profile notable and
distinct?  If that's the case, multiple channel pools are necessary.

> If I don't do that then I have to layer another addressing mechanism
> on top of that provided by BEEP. BEEP, itself, associates channels
> with message purpose by assigning channel 0 as the control channel.

If the profile is stateless, but the messages are indistinguishable as
independent entities (i.e. the same data sent on different channels
could have different actions), it sounds like this is a case for more
than one profile.

> to layer on top of BEEP, the less compelling is the argument for using it.

This is a straw man.  The resource overhead of a channel, once
running, is negligible to that of a socket.  BEEP allows up to 2^31
channels per connection; TCP/IP allows only 2^16 ports per address.

> BEEP could do it over a single BEEP channel and save me the bother.

I think the complexity of maintaining a channel pool is not as bad as
you think, but I'm still curious as to what you meant back in point 1.

--Jered

---

From: "Marshall Rose" <[email-masked]>
Date: Thu, 25 Jul 2002 08:38:21 -0700

hi. first, i think the simplest solution is to simply define a beep feature
that gets negotiated during the greeting that relaxes the serial processing
requirement.

> still performed serially, right?

true. but i was perhaps too subtle in my explanation. darren's use of
MSG/ANS in the syslog profile is rather inspired: the server sends a MSG and
the client streams back zero or more ANS.

now, if i wanted to do your application with a vanilla beep engine, i would
have each side send a MSG to the other. i would then allow each side to put
their requests and responses in the ANSs that get sent back. although
there's an extra level of wrapping here (you put request numbers in the data
you send and the BEEP msgno's are opaque), you can implement whatever policy
you want all over one channel...

> channel 0 as the control channel.

i'm not really sure i understand the concern, even though i agree with the
first sentence. you can certainly define a profile that says each MSG/RPY
contains an opaque blob (or whatever you want).

> to layer on top of BEEP, the less compelling is the argument for using it.

firewalls, redirectors (IP address shifting), NATs, etc.

> BEEP could do it over a single BEEP channel and save me the bother.

i agree. hence your original comment about defining a new BEEP feature...

/mtr

---

From: Darren New <[email-masked]>
Date: Thu, 25 Jul 2002 08:46:59 -0700

Paul Andrews wrote:
> the slowest response. 

Then the client peer can send a MSG with the request and receive a NUL in
response, and then the server can send a MSG as a response and the client
can send a NUL in response.

-- 
Darren New 
San Diego, CA, USA (PST). Cryptokeys on demand.
** http://home.san.rr.com/dnew/DNResume.html **
** http://images.fbrtech.com/dnew/ **

Things to be thankful for, #37:
   No sausage was served at the Last Supper.
_______________________________________________
BEEPwg mailing list
[email-masked]
http://lists.beepcore.org/mailman/listinfo/beepwg

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 11:52:04 -0400

Thanks for taking the time to think about this, and hopefully I'm not being
too troublesome (or just plain dumb) but there are few people around that I
can work through these sorts of issues with.

I've been trying to think of an example that would illustrate what I am
driving at and I think that a web browser loading a web page works quite
well - it is familiar to everyone and it illustrates what I am trying to say
from the point of view of a client.

So. When a web browser loads an HTML page, that page will have on it many
references to objects that also need to be retrieved in order to fully
render the page. Generally, the layout of the page is known from the HTML
document - all the render has to do is retrieve the various referenced
objects and render them into the space that has been left for them. That, to
me, is the interesting part. A browser parallelises the retrieval of those
objects by opening multiple TCP connections and issuing HTTP/GET commands
down each one. Each connection itself operates serially.

The analogy in BEEP would be to create a profile that specifies the use of
GET. The browser then uses multiple channels to download the multiple
objects. Intra channel communication being serial, inter channel
communication being parallel.

The browser now has essentially the same management problem that it had with
plain old TCP. It has to decide when to create a channel and when to release
it to avoid the overhead of channel creation. It has to queue requests to
each channel to ensure that the requests are written atomically (actually,
that's a supposition). If one response on a channel takes a long time (maybe
the server is drawing a graph into a JPEG and then returning the JPEG)
subsequent responses on that channel will be held up and the page renderer
will have to wait.

On the other hand, if the browser could just use one channel (the GET
channel), down which it could pump requests and from which it could retrieve
responses in any order, the renderer would not be held up (at least not by
the communication protocol). There would be less management overhead - just
the one channel to decide to open and release. There would be no contention
for the use of channels (either writing to them or reading from them).

OK, so the renderer would have to decide just how many objects it was going
to try and draw in parallel, but that decision would be independent of how
it uses a protocol to retrieve the objects it wants to draw. In fact, it
might have a thread pool that it allocates rendering operations to. But the
point is (maybe) that there is only one pool to have to worry about, at
least as far as the browser is concerned.

I guess the point is that the restriction on ordering of responses seems to
be arbitrary when it comes to the actual protocol definition itself. If a
given library feels that it has to impose that restriction, then that should
be up to the library. Though I guess that both ends would have to agree on
any ordering restrictions as part of the session negotiation.

The problem I have on the server side is very similar - i.e. there is an
engine that can process many requests in parallel. Now, I receive multiple
requests on one channel and I could process them in parallel. If the API
will not deliver me a request until it has received a response to an
outstanding request, it is effectively throttling me. Maybe the API will
deliver me multiple requests but relies on me returning response strictly in
order - now I have to throttle myself. If the API will deliver multiple
requests and accept responses in any order then it will have to throttle
itself in order to meet the serialization requirement.

OK the client could use multiple channels, but then the client is
effectively dictating how many operations the server can perform in
parallel.

> Sent: Thursday, July 25, 2002 10:21 AM
> To: Paul Andrews
> Cc: Marshall Rose; [email-masked]

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 12:02:33 -0400

First of all, thanks for your considered reply. Its difficult to find people
I can have this level of conversation with so forgive me for delving further
into this below...

> Sent: Thursday, July 25, 2002 10:01 AM
> To: Paul Andrews

That's a good point that hadn't really sunk in for me. In fact the SOAP over
BEEP example does precisely that, i.e. binds a particular SOAP message to a
profile, then you can get multiple channels bound to that profile.

One way of looking at that is that there is a virtual channel that is the
sum of all channels that share a profile.

> data dependencies by pipelining requests on a single channel.

I take your point about the benefit of having enforced pipelining for
certain types of problem.

> out-of-order replies (while maintaining correctness).

I agree. But that is not a requirement in my case. Or at least, where it is
a requirement there are fairly simple mechanisms for enforcing ordering that
do not require support from the communications protocol.

> we're back to something that looks like the channel mechanism, though

So are you saying that MSGs are essentially sent serially, regardless of how
many channels are open? i.e. if I am sending a very large message on one
channel, othe channels will have to wait to before they can send anything?

> requests into a server framework.

But I like shooting myself in the foot! I guess the point is that
serialization is something that can be layered on top of parallelism, but
not vice-versa. At least when operating within the constraints if a single
framework.

> it, though.  (Hm, and I really must roll the 0.9 release soon; I meant

Interesting point. I guess I tend to think that standards are written in
stone. I could act unilaterally to remove the serialization requirement but
I guess I need to convince myself that I'm not talking rubbish and I would
also prefer to work within the standards process if at all possible.

Once again, thanks for taking the time to think about this.

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 12:15:34 -0400

OK. I don't want to rat-hole on this so: do I understand correctly that you
are proposing to amend the RFC to make serialization on a channel optional?

The rest of my points below all boil down to the same thing (I think).
Namely that it would be a shame to have to layer another communications
protocol on top of BEEP. Naturally I'd be layering an application protocol
on top, but then that's the whole point.

By the way if we end up using BEEP in our projects I'd be more than happy to
get involved either at the standards level or at the library level assuming
that no-one objects of course (and assuming that my boss didn't mind and our
legal department etc. etc.).

Thanks for taking the time to reply by the way.

> From: Marshall Rose [mailto:[email-masked]]
> Sent: Thursday, July 25, 2002 11:38 AM
> To: Paul Andrews; [email-masked]
> Cc: Marshall Rose

---

From: Darren New <[email-masked]>
Date: Thu, 25 Jul 2002 09:19:00 -0700

Paul Andrews wrote:
> render the page.

...

> communication being parallel.

The browser could send one MSG with a GET listing *all* the URLs it wants to
retrieve from that server. The server could put each response into a
separate ANS. The ANS can be returned in any order, interleaved. However,
the browser would not get any responses to the second GET until all
responses to the first GET had been received. (It would also be difficult to
abort cleanly without dropping the TCP connection.)
 
> If the API will deliver multiple

This is how the core of the BEEP-C library works. If you reply out of order,
your reply is just queued in the library until you have everything in order.
You can even process and queue chunks in interleaved order, with the first
response being picked out of the stream, then the second response being
sent.

> channel, othe channels will have to wait to before they can send anything?

No, he's saying that if you're allowing multiple MSGs being processed at
once, then the next logical step is to allow multiple MSGs to be sent
concurrently. Otherwise, you're holding up the second MSG while waiting for
the first MSG to go out over the wire. If the first MSG is very large and
the second MSG is small and fast to process, you might be able to get the
response from the second MSG before you even finish sending the first MSG.
But that's not how it works now, because once you start sending the first
MSG, you have to finish sending it.

Right now, you can start sending a MSG on a second channel before the MSG on
the first channel is finished.

> I would

You are working within the standards process. Look at the name of the
mailing list. :-)

---

From: "Marshall Rose" <[email-masked]>
Date: Thu, 25 Jul 2002 11:25:11 -0700

> can work through these sorts of issues with.

no. this is a good discussion as it illustrates a lot of issues...

darren pretty much explained the issues in his reply, in particular the
so-called head-of-line problem in trying to move multiple things on one
channel. this leads us to:

[ the web browser example ]

> effectively dictating how many operations the server can perform in

the whole point here is that client should be using multiple channels
because tit should want to manage the html and the gifs with different
priorities!

/mtr

---

From: "Marshall Rose" <[email-masked]>
Date: Thu, 25 Jul 2002 11:28:06 -0700

> are proposing to amend the RFC to make serialization on a channel
optional?

no. what i'm proposing is that we use beep's extension mechanism (called
"features") and write a new document that talks defines a feature that
removes the same-order property. rfc3080 doesn't change. presumably a new
rfc gets published that describes how something implementing rfc3080 can
also say that it does the new thing too.

/mtr

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 15:02:45 -0400

> Sent: Thursday, July 25, 2002 2:25 PM
> To: Paul Andrews; Jered Floyd
> > too troublesome (or just plain dumb) but there are few people
> around that
> I

Yes. So it uses one channel for HTML and one channel for all of the GIFs it
wants to download in parallel. Or is that what you're saying?

---

From: Jered Floyd <[email-masked]>
Date: 25 Jul 2002 15:25:00 -0400

"Paul Andrews" <[email-masked]> writes:

> First of all, thanks for your considered reply. Its difficult to
> find people I can have this level of conversation with so forgive me
> for delving further into this below...

You're welcome... there's no need to apologize; as Darren pointed out,
you're fulfilling the purpose of this list.  And it's always useful to
have more people with an understanding of BEEP, so that it can take
over the world. :-)

> One way of looking at that is that there is a virtual channel that
> is the sum of all channels that share a profile.

As long as the channels are stateless, yes.  It's a nice abstraction
to have if your application wishes request parallelism.  You can also
use channel groups, or the pipelining on individual channels, to
provide the same sort of functionality as SCSI's tagged queuing.  "I
want all of these tasks done as quickly as possible, and here are a
few ordering constraints."

> > Question: If we remove the restriction on reply-ordering, do we
> > then also allow frames on multiple RPYs or ERRs to be interleaved?
> > That seems like the logical next step.  Of course, then for
> > parallelism we really ought to allow frames of MSGs to be
> > interleaved.  But, then we're back to something that looks like
> > the channel mechanism, though less explicitly managed!
> 
> So are you saying that MSGs are essentially sent serially,
> regardless of how many channels are open? i.e. if I am sending a
> very large message on one channel, othe channels will have to wait
> to before they can send anything?

Not quite, MSGs are sent serially on a particular channel, but frames
on multiple channels may be interleaved.  If we remove the
restrictions on RPY ordering, it seems most sensible to also allow
MSGs and RPYs on the same channel to be interleaved, and then I think
we've gone and lost most of the advantages of having multiple channels
in the first place.  (That is, we get something that is confusing and
complicated.)

> not vice-versa. At least when operating within the constraints if a single

Metatarsal ballistics aside, while we can implement serialization on
top of a parallel layer, we can't implement pipelining on top of a
parallel layer (because we have to wait for the response to arrive on
a channel before we know we can send the next request and have it be
processed in order.)  I think one of the purposes of the channel
abstraction is to explicitly allow for both the intra-channel
asynchorony of pipelining and the inter-channel asynchrony of
parallelism.  I'd rather not see pipelining go away in favor of
intra-channel parallelism... I don't think it's called for.

(I won't address your web browser example here, as I think Darren and
Marshall did a good job of that.)

> also prefer to work within the standards process if at all possible.

I'd be happy to point you at the right parts of the code if you'd like
to explore this further, but I really do think that channel pools are
the right way to go about doing this.  But perhaps I'm old and set in
my ways. *grin*

--Jered

---

From: Jered Floyd <[email-masked]>
Date: 25 Jul 2002 15:36:08 -0400

> wants to download in parallel. Or is that what you're saying?

Not necessarily; I think Marshall was suggesting that with a set of
indistinguishable channels, it uses some subset (say, one) for
retrieving HTML, and another subset (as many as it wishes, or thinks
it can handle in parallel), for retrieving the GIFs.  It can always
hold onto more channels than it is using at any given time, as
channels are lightweight and cheap.

Consider this to be the same way existing browsers manage multiple
connections, except that the resource cost is not as steep.

--Jered

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 16:35:51 -0400

> From: Jered Floyd [mailto:[email-masked]]
> Sent: Thursday, July 25, 2002 3:25 PM
> To: Paul Andrews
> processed in order.)...

Weeeelllll. If I understand correctly (a big if at this point). You still
don't have that guarantee because all you are specifying is that replies
have to be sent in the same order that requests are received. Nothing is
stated about the order in which requests have to be processed. In fact to
paraphrase an earlier reponse in this thread: the library actually holds
responses until all repsonses to eariler requests have been sent. This
implicitly allows you to give responses to the library out-of-order.

> ... I think one of the purposes of the channel

It would be nice to be able to negotiate the exact level of synchronicity
(synchronousnous?) required on a single channel.  i.e.:

- Totally asynchronous.
- Synchronous responses.
- Synchronous requests and synchronous responses.

In fact I think Marshall has pointed out that that is a possibility using
'features'.

This would not obviate the need for channels, it would just free them up to
be used at a purely application semantic level rather than a mix of
application and protocol semantic levels.

> the right way to go about doing this.  But perhaps I'm old and set in

LOL. Bet you're not as old as me.

Anyway, yes you can point me at the right parts of code. I'm a Java
programmer by preference. Of course, I might come to the same conclusions as
y'all.

By the way, why isn't PermaBEEP listed at beepcore.org?

---

From: "Paul Andrews" <[email-masked]>
Date: Thu, 25 Jul 2002 16:44:31 -0400

> Sent: Thursday, July 25, 2002 2:28 PM

Boy. Talk about jumping in at the deep end (to mix my metamphors). Just when
I thought I was getting a handle on all this BEEP stuff I'm introduced to
features. I have to say I'd tried to find out how the 'Extensible' bit
worked. I guess I should've read the RFC from start to finish.

I don't suppose you have an 'Idiots Guide to Adding Features' do you?

---

From: Jered Floyd <[email-masked]>
Date: 25 Jul 2002 17:26:48 -0400

> stated about the order in which requests have to be processed. 

It's not as clear as I'd like, but Section 2.6.1 of RFC 3080 says:

   A BEEP peer acting in the server role must process all "MSG"
   messages for a given channel in the same order as they are
   received.

I suppose this could be taken to mean that the BEEP layer, and
not necessarily the application, need do so, but I don't like
that interpretation.

> In fact to paraphrase an earlier reponse in this thread: the library
> actually holds responses until all repsonses to eariler requests
> have been sent. This implicitly allows you to give responses to the
> library out-of-order.

BEEPCore-C does this.  This moves the responsibility for adhering to
S2.6.1 to the application.  We chose not to do this in PermaBEEP to
protect developer toes.

> programmer by preference. Of course, I might come to the same
> conclusions as y'all.

To remove the restriction on delivery of MSGs to the application, I
think it should just be a matter of removing a check in
Channel.getMessage() (and perhaps cleaning up assorted unneeded
notifications.)  How do provide out of order RPYs (non-interleaved)
actually requires a bit more thought from me, but all of the relevant
modifications for everything we've discussed would be to Channel.java.

> By the way, why isn't PermaBEEP listed at beepcore.org?

It's listed under Products, not Projects.  I suppose I should create a
duplicate entry, but embarassingly I've forgotten my beepcore login
password.  Who's in charge of the site these days?

--Jered