Subject: [Rserpool] Re: Message channel issues

From: Qiaobing Xie <[email-masked]>
Date: Tue, 11 Mar 2003 17:00:53 -0600

Hi, all,

When we started to consider defining a control channel at our interim
meeting (1.5 year ago?), it was thought to be a small addition to the
architecture that would allow the passing of a few optional messages
(buz-card, cookie, etc.). The use of these messages were more meant to
somehow enhance the functionality of RSERPOOL. Now it seems that this
creature is growing rapidly in terms of architectural impact and
complexity. Maybe it's time for us to do a sanity check and re-visit how
much we can gain from this and what we are going to pay for it.

>From the design's perspective, the main thing in the current proposal is
to try to bring the control channel from PE-PU communication link into a
separate entity. This IMHO could have a big implication to RSERPOOL and
raises a lot of open issues. In the following, I will pose a serious of
questions for us to think about.

"Randall Stewart (cisco)" wrote:
> 
> Hi all:
> 
> I was asked to summarize the message channel issue and put out
> one of the solutions (and one other issue) we discussed on yesterdays
> conference call.
> 
> -------------------------
> Issue #1
> -------------------------
> The basic issue is this:
>

Keep in mind that currently we allow a PE to register with 1 or more
transport addresses of diff types (e.g., a PE is _allowed_ to reg with
sctp-trans1, sctp-trans2, tcp-trans, udp-trans1, udp-trans2), and we
have of cause 1 or more PEs in any given pool.
 
> 1) A PE may register a Control channel, Data Channel or Both (in a combined
>     form or seperate).

- A founding principle of RSERPOOL is that all PEs in a pool are
"functionally identical" while "maybe capability-wise different". If we
start to introduce "control-only PEs" via "data-only PEs" via
"control/data PEs" in the same pool, we need to understand the
implications and why we need this complexity. For example, can you
failover from a control-only PE to a data-only PE? What does that mean?

- How does the load sharing should be understood in this picture? I.e.,
can one round robin amongst a mixed set of data-only and data-control
PEs? What's the implication here?

- Are we going to allow a PE to reg as "control channel only"? I'd
strongly advice against that since it seems to be meaningless and should
be disallowed, IMO. After all Rserpool is in the business of moving user
data and I don't see why a PE is useful being control-only.

- It must be made clear that unreachability detection must be tied to
the data side. Otherwise, the redundancy becomes meaningless (e.g., what
if a PE becomes unreachable on its data side while its control side is
still functioning?). Question is how to make sure this happens?

- In a related issue, what if a PE become unreachable over its control
channel while its data channel is ok? We should define what this means.
Should we monitor reachability of control channel at the first place? If
so, how?

- When a PE registers with >1 transports (of maybe mixed types among
SCTP, TCP, UDP), how the control and data channel map to that? Can it do
things like reg[sctp1/control-only, sctp2/control-and-data,
tcp/data-only]? If not, what'are the restrictions and how to enforce
them? If allowed, how those mixed PEs are going to behave in terms of
failover and load sharing (see below also)?

- How the failover will work now? Can a PU failover from a control/data
PE to a data only PE? Or, should we force all PEs in a pool to have the
same control/data config? 

I can go on and on with this type of questions... there are so many
combinations here. And I can imagine it's going to be very expensive to
design and implement them all in RSERPOOL. We really need to understand
why we need each of them before we decide to build it into RSERPOOL. 

As I mentioned above, most of the issues above arise from the fact
that we are now trying to bring the control channel from PE-PU
communication link into a separate entity. One alternate approach would
be to keep the control channel bundled with data channel. This would
solve a lot of issues mentioned above. By this, I mean basically that we
keep our focus on data side and only allow to have a control channel if
the data side is over SCTP (using PPID to mux) or TCP (using mapping to
mux). This is simply and its impact to architecture is little to none.
This also automatically solves the problem of tying control and data.

regards,
-Qiaobing

> 2) If the PE register's a Data channel and a control channel (non-combined) then
>     how does the PE make the association between the PU talking on the two
>     differerent channels?
> 
> Suppose The PU associates with the Data Channel of a PE... Then at
> some point decides to use the Control Channel.. for the PE a message
> arrives on the Control channel and there is no tie back to the conversation
> on the data channel?
> 
> Or suppose the control channel is first associated with, followed by
> the data channel.. how does the PE make the connection..
> 
> One of the proposed solutions we discussed and thought worth considering (to make
> sure we have covered all the bases) is:
> 
> 1) Require that if a PE has both a control and data channel that the PU ALWAYs associate
>    to the Control channel and send a Bussiness card specifying itself.
> 
> 2) The business card will need to be changed so that we can identify that this is a PU only
>    address.
> 
> 3) The PU would need to supply all of the addresses to the PE for its side of the Data channel so
>    if it has two communication endpoints up (SCTP/control and TCP/data for example) it would place
>    in its B-C a indication of the TCP channel id (port/ip address etc) and mark it as "Data channel
>    only, will not accept connections". The SCTP information could be deduced from the
>    association.
> 
> With this proposal we would eliminate the new message that was just added. Expand the
> types that was just added to asap (for TCP and SCTP endpoints) and finally enhance the
> wording of the business card and require that ... if a Control channel exists the PU MUST
> connect and pass its B-C... or something like that.
> 
> ---------------
> Issue #2
> ---------------
> 
> The basic issue:
> 
> There are many possiblities of combinations of Data and Control
> channels.. do we need to have some sort of restrictions on these... here is a table I have
> put together showing the possibilites
> 
> Control Channel Type          |             Data Channel Type
> ------------------------------+--------------------------------------------------------
>   SCTP                        |               SCTP - Multiplexed - same association
> ------------------------------+--------------------------------------------------------
>   SCTP                        |               SCTP - Seperate assocaition
> ------------------------------+--------------------------------------------------------
>   SCTP                        |              TCP
> ------------------------------+--------------------------------------------------------
>   SCTP                        |               UDP
> ------------------------------+--------------------------------------------------------
>   TCP                         |               SCTP
> ------------------------------+--------------------------------------------------------
>   TCP                         |               TCP - Seperate connection
> ------------------------------+--------------------------------------------------------
>   TCP                         |                TCP - Multiplexed - same connection
> ------------------------------+--------------------------------------------------------
>   TCP                         |               UDP
> ------------------------------+--------------------------------------------------------
> 
> So we basically have eight different combinations.. if I have gotten them all
> (remember UDP is not allowed for the control channel)...
> 
> So the question is.. should some limitations be placed on the allowable combinations?
> 
> Thanks
> 
> R
> 
> --
> Randall R. Stewart
> ITD
> Cisco Systems Inc.
> [email-masked] 815-342-5222(cell) or 815-477-2127(office)

_______________________________________________
rserpool mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rserpool

---

From: Qiaobing Xie <[email-masked]>
Date: Tue, 11 Mar 2003 17:00:53 -0600

Hi, all,

 

regards,
-Qiaobing

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 05 Mar 2003 11:05:08 -0600

Hi all:

1) A PE may register a Control channel, Data Channel or Both (in a combined
    form or seperate).
2) If the PE register's a Data channel and a control channel 
(non-combined) then
    how does the PE make the association between the PU talking on the two
    differerent channels?

One of the proposed solutions we discussed and thought worth considering 
(to make
sure we have covered all the bases) is:

1) Require that if a PE has both a control and data channel that the PU 
ALWAYs associate
    to the Control channel and send a Bussiness card specifying itself.

2) The business card will need to be changed so that we can identify 
that this is a PU only
     address.

3) The PU would need to supply all of the addresses to the PE for its 
side of the Data channel so
     if it has two communication endpoints up (SCTP/control and TCP/data 
for example) it would place
     in its B-C a indication of the TCP channel id (port/ip address etc) 
and mark it as "Data channel
     only, will not accept connections". The SCTP information could be 
deduced from the
     association.

With this proposal we would eliminate the new message that was just 
added. Expand the
types that was just added to asap (for TCP and SCTP endpoints) and 
finally enhance the
wording of the business card and require that ... if a Control channel 
exists the PU MUST
connect and pass its B-C... or something like that.

---------------
Issue #2
---------------

The basic issue:

There are many possiblities of combinations of Data and Control 
channels.. do we
need to have some sort of restrictions on these... here is a table I have
put together showing the possibilites

Control Channel Type  |             Data Channel Type
------------------------------+--------------------------------------------------------
  SCTP                           |               SCTP - Multiplexed - 
same association
------------------------------+--------------------------------------------------------
  SCTP                           |               SCTP - Seperate assocaition
------------------------------+--------------------------------------------------------
  SCTP                           |              TCP
------------------------------+--------------------------------------------------------
  SCTP                           |               UDP
------------------------------+--------------------------------------------------------
  TCP                             |               SCTP
------------------------------+--------------------------------------------------------
  TCP                             |               TCP - Seperate connection
------------------------------+--------------------------------------------------------
  TCP                             |                TCP - Multiplexed - 
same connection
------------------------------+--------------------------------------------------------
  TCP                             |               UDP
------------------------------+--------------------------------------------------------
 
So we basically have eight different combinations.. if I have gotten 
them all
(remember UDP is not allowed for the control channel)...

So the question is.. should some limitations be placed on the
allowable combinations?

Thanks

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 13 Mar 2003 06:36:49 -0600

> questions for us to think about.

And has you know Q, the Buisness card has been in the orginal
Motorola prototype since before 1999.. because it was esential
when two PE's were speaking to each other (one as PU the other
as PE)... this is nothing new.. I will admit Michael's state
cookie is new... but I don't see it as a huge amount more...

Some more comments...

> have of cause 1 or more PEs in any given pool.

Yep... but we have no classification of what can be sent on
those channels...

> failover from a control-only PE to a data-only PE? What does that mean?

And the B-Card was a founding principle from way back as well.. And I 
don't think you understand the idea here... In my original thoughts you
MUST have one of your registered addresses to accept DATA.. the fact
that you have multiple channels is why I added the classification..

Of course Michael T was thinking about it in more of a ... you
register a control only and then talk to the PE to get the "dynamic"
data channell.. sort of like FTP... Don't know which is best but I
see nothing wrong with classifications on how an address can be
used once you introduce multiple address registries...

> PEs? What's the implication here?

Again it would depend... if we go with Michaels view, you would have
to contact each of the servers to get a data channel... my throughts
were always that the PE had to have a data channel registered (and
possibly control as well.. if it could accept B-Cards/State-cookies).
load sharing would be no different than it is now... you load
share amongst the multiple PEs...

> data and I don't see why a PE is useful being control-only.

This was Michaels concept... think FTP where you talk to the
PE on the control channel and get a data channel from it... now
I am not sure on this... but it is how a large number of applications
do currently work in the big-i

> still functioning?). Question is how to make sure this happens?
This is a major problem with allowing multiple address registrations.
Think about it.. if I allow SCTP-T-1, SCTP-T-2, TCP-T-1 ... what
does it mean (forgetting the data/control thing) if TCP-T-1 becomes
unreachable, while SCTP-T-2 and SCTP-T-1 are ok? Same issue... no
matter if you have a "data type channel" or ones marked that can
be control and data... just control or just data...

I think the data only channel concept is more for older apps aka
those that are proxy'ed in...

In fact I wonder if the data-control designation and data designation
are the only valid ones... i.e. if you register with data-control
you can take b-cards etc.. if you register data then you can
accept no control messages and you can't send any...

Might be a workable arrangment.. but I want to think on this one...

> Should we monitor reachability of control channel at the first place? If

Or reverse the case...

Or for that matter, how does a application detect unreachable data
channels (if they say are UDP?)...

All issues that need to be thought on...

> failover and load sharing (see below also)?

I agree it is in effect a side effect of letting multiple registrations
of different transports be done...

Maybe the solution is no more than two transports be registered
and they must either be

1) A single transport for control and data
2) A single transport for data only
3) Two transports one for control, one for data..

If you select <3> then if EITHER channel does not function
then the PE is considered unreachable..

This better scopes the issue...

Note: now please don't reply with but I want to have an app which does
SCTP and TCP for data... I have always been skeptical of multiple
transport registration... I can see, maybe, having data and
control seperate on different transports .. aka UDP/SCTP
or some such.. but I have a issue with more than
two for sure..

> same control/data config? 

You have the same issue if you allow multiple transports..

Can a PU fail over from a TCP PE to a SCTP PE? I think
the thing is those in the PE should ALL use the same
transports... IMO

> why we need each of them before we decide to build it into RSERPOOL. 

This was the whole point of the email.. we have to come up
with a bounded set of limits to what is allowable...

> communication link into a separate entity. 

No, these issues showed up the minute you allow multiple transport
registrations for a PE.. period... the same issues are present
we just have not thought of them...

One alternate approach would
> This also automatically solves the problem of tying control and data.
And this goes along the lines of my earlier statement above..

Data is always implied, and you can add control to data on
your registration... but this also will remove a possible proxy
type situation .. which may be ok..

I also think you still have these same issues with mulitiple transport
registrations.. instead maybe only 1 type is allowed for a whole
PE...

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Randall Stewart <[email-masked]>
Date: Thu, 13 Mar 2003 06:36:49 -0600

Some more comments...

Or reverse the case...

R

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 13 Mar 2003 13:05:45 -0600

Hi, Randy,

I think we are in general agreement here. I too believe that we may have
allowed too much freedom into the design lately (such as the wide range
of combinations in number and types of transports registration,
combinations in data/control config...). I mean, freedom is good, but as
we all know too well that it does not come free. The cost we may end up
paying could be both in high impl complexity and loss of efficiency and
real-timeliness. More my comments in-line.

Randall Stewart wrote:
...
> cookie is new... but I don't see it as a huge amount more...

I am not questioning about their usefulness. But we need to do it in the
right way.

...
> MUST have one of your registered addresses to accept DATA.. 

Absolutely agree. It is crucial, see below for more on this.

> the fact

Believe it or not I'd love to see that multiple address reg go away :-)

> do currently work in the big-i

Here we must not lose our original design focus here... RSERPOOL is not
originally invented for apps like FTP. It will work well to enhance apps
like FTP, but RSERPOOL's primary application area is to support fast
session fail-over for **real-time or near-real-time apps**. This means
things that work well for FTP type of apps may not satisfy RSERPOOL
requirements (as stated in RFC3237). This is precisely why I've been
constantly worried about complexity. IMO keeping it simple enhances our
chance of reaching that goal. 

> be control and data... just control or just data...

With the restriction we've put in - "multiple transports are only for
initial connection establishment, but NOT to fail-over from one
another", I can at least understand how this works. But again, I'd love
to see the multiple addr reg go away :-)

> those that are proxy'ed in...

Agreed.

> accept no control messages and you can't send any...

This is for older apps. If they are incapable or unwilling to have a
control channel, they will simply not enjoy the benefit of b-card,
etc.... which can be an incentive for them to upgrade :-)

> channels (if they say are UDP?)...

Data channel outage is easier to deal with... in the sense that the app
is always involved in the detection. In the case of UDP, since the app 
is the ultimate sender/receiver of UDP data, if so desired it will be
able to tell (ie, detect) the loss of data channel. But the control
channel is our own thing and is invisible to the app, this means that if
RSERPOOL doesn't do the detection, no-one else will. This is why I think
we need to understand whether and how this detection should be
performed. Maybe after some analysis we conclude that it's not a big
deal losing control channel. But we need to understand it. Moreover, if
we move to the direction of making data-control always bundled together,
this issue would go away.

> then the PE is considered unreachable..

I like this... I'd even go a bit further to take out <3>.

> or some such.. but I have a issue with more than

I, too, have been skeptical of multiple transport registration. It will
simplify a lot of things if we remove it.

> registrations.. instead maybe only 1 type is allowed for a whole

I'd like to see if we can reach some violent consensus to get rid of the
multiple addr reg. Then our data-control discussion would become much
easier.

-Qiaobing

---

From: Qiaobing Xie <[email-masked]>
Date: Thu, 13 Mar 2003 13:05:45 -0600

Hi, Randy,

Randall Stewart wrote:
...

...

Agreed.

-Qiaobing

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 17 Mar 2003 00:15:37 -0500

Some comments:

>I like this... I'd even go a bit further to take out <3>.

Believe it or not, I'm in agreement with Qiaobing on several 
points.   First, I agree that we can probably live without multiple 
transport registrations.   Why did we want these in the first place?   For 
two reasons I think.

(1) We wanted to allow for migration from TCP to SCTP for pool users; i.e. 
allow some pool users that have not yet adopted SCTP to participate in a 
pool for service X.   However, this can be handled more simply by 
maintaining separate pools, with separate pool handles, e.g. X-SCTP and 
X-TCP.   This preserves the principle that a pool should be made up of 
equivalent services... this concept should include "equivalent in terms of 
which transport protocol is used".

(2) We wanted to allow for separate data and control channels on two 
different protocols.  E.g. control channel over SCTP (or mapped TCP) and 
data channel over UDP, or protocol XYZ, or whatever.    However, Qiaobing 
makes a case that we can acheive better simplicity by simply requiring that 
applications that use a transport other than SCTP or TCP for their data 
channel do NOT benefit from the extended features that the control channel 
allows.  I.e., you should either use a multplexed control/data channel, or 
you shoudl not use a control channel at all.   (Interestingly, this is 
pretty much exactly what I had in mind when I suggested two "modes"  in the 
original version of the services draft.    I think perhaps it is time to 
take a second look at that concept.  Qiaobing doesn't seem to like the 
terms "failover" vs. "nameservice" mode  for the services you get from a 
muxed control/data channel, or a data only channel (respectively), but I 
think that the concepts in the services draft are the right ones.

Let me invite folks to re-read the services draft and make some 
suggestions: is the problem with the concepts, or only with the 
terminology?     Can we come up with better terms that "failover" vs. 
"nameservice" mode?

Thanks,
Phill

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 17 Mar 2003 00:15:37 -0500

Some comments:

Thanks,
Phill

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 17 Mar 2003 11:32:06 -0800

Here are some suggested directions for RSerPool:

To the working group: do we have rough consensus on these points?  I and 
others (Peter, Randy, Qiaobing et al)  will ask for rough consensus on 
these points in the working group meeting later today in San Francisco... 
and for the sake of those that are NOT in the meeting today, please 
consider this email as your opportunity to express support, dissent, or to 
ask for further clarification on the points below.

(1) In the service draft, replace "failover mode" vs. "nameservice mode" 
with "multiplexed data channel" vs. "non multiplexed data channel".

(2) In the service draft, remove application layer acks from protocol, and 
write up as use case instead.

(3)  In general: No multiple transport registrations: you either

(a) register the transport address of either an SCTP or TCP (but not both) 
multiplexed control and data over a single transport 
association/connection.  (Note: you _can_ register both, but if you do so, 
they should be in different pools, preserving the principle that all PE's 
in a pool should be functionally equivalent, including all using the same 
transport protocol to provide service.)

(b) you register the transport address of a data channel only, not managed 
by rserpool (and do without the services that would be provided by a 
control channel.)

(4) A Note: when you register for SCTP, you may register multiple IP 
addresses, but only a single port... i.e, you are only registering a single 
SCTP endpoint.  This is already reflected in the current drafts.

Thanks,
Phill Conrad

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 17 Mar 2003 11:32:06 -0800

Thanks,
Phill Conrad