Subject: [Rserpool] Another data channel issue: failover of messages in basic mode?

From: "Phillip T. Conrad" <[email-masked]>
Date: Tue, 01 Apr 2003 17:03:52 -0500

Dear Colleagues:

In my discussion with Qiaobing about renaming the two modes for RSerPool 
services to "basic" and "enhanced", another issue that Qiaobing raised is 
whether in basic mode, automatic failover of messages can be provided as a 
service.

My first inclination is to say "no", however on further reflection, I said 
I would think about it.  So here are some thoughts about that, and an 
question to pose to the working group.

My first reaction is to say that what you get in basic mode is the 
selection of a server that is better than what you would get with DNS 
because it

- is based on reports from PEs and PUs about server health
- can based on a policy value (for load sharing)

(and maybe some other benefits that the working group can help me to 
recite; I'm definitely inviting help with the text here!)

But in basic mode, you do not get automatic best-effort message failover, 
because that requires you to be sending over a transport that is mapped to 
the RSerPool API.   Message failover comes in enhanced  mode, along with 
all the other goodies (business cards, etc)

However, Qiaobing pointed out, correctly, that you do not need the control 
channel to do message failover (now that application layer acks are gone); 
all you need is someway to tell the RSerPool API how to map the the 
RSerPool data transmission primitives (e.g. the equivalent in your 
particular API of the data.send primitive) to the transport protocol in 
question (whether that be SCTP, TCP, UDP, DCCP or whatever).   This mapping 
is so trivial that it almost doesn't even need to be mentioned, and if we 
provide it, this will allow failover of any message that has not yet been 
put on the wire.

Of course, except for with SCTP, you will not get any message failover of 
messages that have already been submitted to the transport layer and were 
never transmitted, since no protocol other than SCTP supports "retrieval" 
(note that because application layer acks were removed, you don't even get 
that with the TCP mapping layer anymore.)

My question is this: how often would it be, for example, in UDP, that you 
would have any messages that had not yet been put on the wire?     The only 
time I can imagine an RSerPool implementation not putting something on the 
wire immediately is if the underlying transport layer socket is not 
writable (determined for example via a select() system call in the sockets 
API, or an EWOULDBLOCK error, etc.)      My understanding is that 
this  typically happens when the transport endpoint is restricted because 
of cwnd or rwnd, and then also runs out of buffer space, and the socket 
runs out of buffer space too.

This could realistically happen with SCTP or TCP particularly if the 
application on the peer end is slow to read and process data.  But how 
realistic is it for UDP?   My guess is that is will never happen, since UDP 
is never blocked due to cwnd or rwnd.

So, I while I can concede Qiaobing's point that a message channel is not 
necessary to provide I'm not sure there is a practical need to provide for 
message failover in basic mode.    I'd like to keep basic mode as basic as 
possible, just for cleanness of the architecture.

I'd like to invite the rest of the working group to comment on these two 
possibilities: should we

(1) allow for message failover in basic mode for messages not yet put on 
the wire (which means, conceivably, any arbitrary transport protocol)?

OR

(2) restrict message failover to enhanced mode (SCTP and mapped TCP)?

Thanks,
Phill

_______________________________________________
rserpool mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rserpool

---

From: Qiaobing Xie<[email-masked]>
Date: Wed, 02 Apr 2003 18:07:44 -0600

Hi, Phill,

Please see my comments below...

"Phillip T. Conrad" wrote:
...
> that with the TCP mapping layer anymore.)

To better describe this, we probably should use 2 separate terms here:

"Session Failover (SF)" - upon a failure of the current PE, the action
of the PU picking up an alternate PE in the pool and carrying on the
comm session btw the pool user and the pool.

"Message Failover (MF)" - upon a failure of the current PE, the action
of session failover, PLUS, in a best-effort fashion, re-sending un-acked
messages to the new PE.

SF is the key and a fundamental objective of RSERPOOL. This is expressed
all over the requirement RFC (all the examples are meant to show this
benefit). It has little to do with the control channel. But how well
session failover can be performed depends on the transport - with well
configured SCTP as transport, SF can be fast and smooth; with TCP, SF
may come with long latency; with UDP, SF can only happen when the app
assists in reachability monitoring.

MF is something desirable but not fundamental to RSERPOOL. This is why
we stated "transaction integrity not guaranteed" when we worked on the
requirement. Similarly, MF has little to do with the control channel. It
mainly depends on whether the transport layer provides data retrieval
services. With SCTP, your chance of having data retrieval is good. With
TCP or UDP, you probably will be out of luck. 

What difference the control channel (as currently defined for Cookie and
B-card) makes is to allow a finer control on the failover transition.
With B-card, you can select the best candidate PE; with Cookie, you can
better prepare the new PE by passing some app state info. But none of
those will play a role in MF. 

If you are talking about TCP mapping in particular, I think we need to
point out that TCP mapping != control channel, nor it means MF. In the
TCP draft, if I read it right, PPID and TSN are both optional. This
means that if I turn off the PPID then there won't be a control channel.
Moreover, if I turn off the TSN, then there won't be an ACK service that
can create MF possibility. 

> possible, just for cleanness of the architecture.

Yes. This (or similar reasoning) was what led many in the requirement
team to believe MF should not be made a hard requirement. E.g., I don't
think there are many UDP apps need MF service.

> (2) restrict message failover to enhanced mode (SCTP and mapped TCP)?

This means, if my PU uses SCTP in basic mode, my PU is disallowed to use
data retrieval provided by SCTP? Why should I be restricted like this?
You can only convince my if you can show me that either doing so is
harmful or it is useless for all applications.

regards,
-Qiaobing

---

From: Qiaobing Xie<[email-masked]>
Date: Wed, 02 Apr 2003 18:07:44 -0600

Hi, Phill,

regards,
-Qiaobing

---

From: Peter Lei <[email-masked]>
Date: Wed, 02 Apr 2003 23:00:08 -0600

Comments inline...

> > that with the TCP mapping layer anymore.)

    As in it is no longer a MUST / SHOULD requirement.  TCP mapping layer
    MAY implement buffering for retrival.

> those will play a role in MF.

Agree up to here...
 
> can create MF possibility.

The TCP mapping draft does state that PPID is required for multiplexed
control and data (e.g. control = ASAP PPID); as of San Fran, only this
multiplexed control and data would be used now anyways, right?  I had
intended on making this change (non-optional PPID) in the next pass.

Also in San Fran, I indicated that we would note in the draft that there
is no requirement for implementing a buffer for message retrieval, but
MAY be desirable; certainly without buffering, MF cannot be performed
(and as you point out, TSN is also needed then).

While the fields are optional, they are meant to be implemented... they
provide the necessary hooks in order to provide for SF and w/buffering,
MF.  Recall that the motivation for the TCP mapping draft, in particular,
was to allow non-SCTP transport; there are additional features like
heartbeating that help SF as compared to plain TCP.

> harmful or it is useless for all applications.

I'm not sure I follow your scenario... the PU is not "disallowed" from
using SCTP retrieval... it's just not "automatically" done on behalf of
the PU by the rserpool framework / ASAP;  the PU is certainly able to
retrieve and re-send "manually" if it wishes upon SF.

thanks,
--peter

---

From: Peter Lei <[email-masked]>
Date: Wed, 02 Apr 2003 23:00:08 -0600

Comments inline...

Agree up to here...

thanks,
--peter

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 03 Apr 2003 14:43:27 -0600

Hi, Peter,

Peter Lei wrote:
...
>     MAY implement buffering for retrival.

That sounds good.

> (and as you point out, TSN is also needed then).

I'd agree with the above.

> heartbeating that help SF as compared to plain TCP.

I understand they are mandatory to impl but optional to use. This is why
I used terms like "turn on/off" in my email. This in fact means that
when a PU using TCP mapping start a session with a pool, it has the
choice of whether to use a ctl-channel or not. I think this flexibility
is good. The point I was trying to make in my earlier email is that
using TCP mapping does not automatically mean that the session will
always have a ctl-channel or MF. 

I think we are basically in agreement here :-)

....
> retrieve and re-send "manually" if it wishes upon SF.

Then I guess I misunderstood Phill. The words "allow" and "restrict" are
strong words in a protocol spec :-) So, it is ok that I can build an
SCTP-based app that chooses not to use a ctl-channel, but flag the stack
to perform best-effort MF during fault recovery? If so, I am happy and I
don't mind what you call this mode. I just don't want to see an impl
that forces me to turn on the ctl-channel whenever I try to flag for
performing best-effort MF. I'd think the implementation should give the
ULP two separate controls (e.g., via ioctl calls) - one for MF and one
for ctl-channel. 

regards,
-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 03 Apr 2003 14:43:27 -0600

Hi, Peter,

Peter Lei wrote:
...

That sounds good.

....

regards,
-Qiaobing

---

From: Phillip Conrad <[email-masked]>
Date: Thu, 03 Apr 2003 23:41:15 -0500

>ULP two separate controls (e.g., via ioctl calls) - one for MF and one

I think that should be possible.   What I am saying is that the way you would achieve this goal is to use enhanced mode (because that is the mode that supports MF)  but just never "use" the control channel.   If a control channel is theoretically present, but no messages ever actually pass over it when your application is running (because your application never "turns on" the extra ASAP services such as business cards that would cause messages to be exchanged over that control channel), is that close enough to "not having a control channel" to satisfy your needs?  

(Note that the upper layer never accesses the control channel "directly"; the control channel is used for transmission of ASAP messages, so there is a layer of indirection between the application and the control channel).

Keep in mind that since we have already decided that we are NOT defining any "subsets" of RSerPool, it wont be the case that a given ASAP implementation "can't understand" a control channel.  It just might be that for some particular application, you don't want the extra bits on the wire that a control channel would impose, or the processing overhead of dealing with a control channel.     That's understandable.   Having a "totally quiet" control channel should be enough to meet that goal, right?

So what I am suggesting is that you want MF, you should use enhanced mode, because that's the mode that supports the data sending primitives that are needed to get MF.  It also happens to be a mode _supports_ a control channel... but if you don't ask for any services (such as business cards, etc) that  require a control channel, then nothing will actually ever "go" over the control channel, and that is no different from the control channel "not being there". 

To emphasize this point, we can perhaps, when we get around to defining APIs, do so in such a way that the default in enhanced mode is that all the things that require a control channel are by default OFF.  That way, enhanced mode by default provides SF and MF, and nothing else, so that the control channel is by default, "quiet" (practically non-existent).    (At least, I think we can do this...  things have been changing so fast, that I will need to re-read everything again before I can be 100% sure of this. :-) ).

Regards,
Phill

---

From: Phillip Conrad <[email-masked]>
Date: Thu, 03 Apr 2003 23:41:15 -0500

Regards,
Phill

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 04 Apr 2003 17:29:23 -0600

Phill,

> I think that should be possible.   What I am saying is that the way you would achieve this goal is to use enhanced mode (because that is the mode that supports MF)  but just never "use" the control…

This would work for me. Strictly speaking, this has more to do with the
implementation strategy. I am fine with this, as long as we don't let
people have the wrong impression that "to do MF one must have a
ctl-channel."

> (Note that the upper layer never accesses the control channel "directly"; the control channel is used for transmission of ASAP messages, so there is a layer of indirection between the application a…

This is a little bit unclear. For example, when a PE decides to send a
B-card with a wish list, where should the PE get that list? I guess it
has to be passed down by the app, right? Another example, when a PE
wants to send a Cookie, it can only do so under the instruction of the
ULP and must get the app state info from the ULP. So I think the app
must know the existence of the ctl-channel and the app must determine
whether to use a ctl-channel at the initiation of the session.

> Keep in mind that since we have already decided that we are NOT defining any "subsets" of RSerPool, it wont be the case that a given ASAP implementation "can't understand" a control channel.  It ju…

Again, I am ok with this. I consider this a reasonable impl strategy.

> So what I am suggesting is that you want MF, you should use enhanced mode, because that's the mode that supports the data sending primitives that are needed to get MF.  It also happens to be a mode…

Ok. I think this is just about terminology. In my example, I would then:
1) be using a rserpool impl that supports ctl-channel, 2) have turned on
MF and therefore in the enhanced mode, and 3) not be using the
ctl-channel. 

> To emphasize this point, we can perhaps, when we get around to defining APIs, do so in such a way that the default in enhanced mode is that all the things that require a control channel are by defa…

Yeah, it's quite confusing now. Let me try this:

basic mode - defined as a PU <-> pool session with no control channel;
enhanced mode - defined as a PU <-> pool session with a control channel;

Note, I don't say PE here because of potential SF. Let's see if we can
agree on this first :-)

regards,
-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 04 Apr 2003 17:29:23 -0600

Phill,

regards,
-Qiaobing

---

From: Randall Stewart <[email-masked]>
Date: Wed, 16 Apr 2003 13:50:45 -0500

I think I have read most of this thread.. this
seems to be the last one.. soo some comments :>

> people have the wrong impression that "to do MF one must have a
I never had the impression that anyone was saying that.. and keeping
things as SIMPLE as possible is ALWAYS a good idea IMO :>

> has to be passed down by the app, right? 

No, the PE is going to get the list itself from its ENRP server. It
gets a list of those registered with its same Pool Handle and then
does a bashing of the list to figure out what order it wishes to
send these to the PU it is talking to. The app may provide
guidance in the form of we are doing a rondevaue or we are doing
a dispersal .. with that information the PE must figure out
what to put in the BC ..

Another example, when a PE
> ULP and must get the app state info from the ULP. 

This is definetly true. aka it must come from the app

So I think the app
> whether to use a ctl-channel at the initiation of the session.

I don't think the above is true.... One can easily design the app
to do

xxx = send_asap_cookie(state-cookie, pe);

if( xxx == -1) {
    /* no control channel this guy will
     * have problems failing over
     */
    ....;
}

This way you find out when you attempt to do the actual
cookie population and maybe take corrective action.

Now on this same subject, I would doubt the above would happen.. after
all if I am writting a PE I would think I would be aware of the services
I am offering my client PU and also what contract I make with him.. i.e.
you must connection to me on a control/data channel over SCTP.. etc.

A good app would have the above ...; to catch a error.. say sending
an alarm or logging a issue.. but in general that code should join
the vast majority of good code... hardly being executed since it
is in the error path :>

> MF and therefore in the enhanced mode, and 3) not be using the

I think this is the right approach.. you keep basic service very very
basic.. and everything else is in the enhancned mode...

> Note, I don't say PE here because of potential SF. Let's see if we can
I would agree with the above :-)

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Randall Stewart <[email-masked]>
Date: Wed, 16 Apr 2003 13:50:45 -0500

So I think the app

if( xxx == -1) {

R