Subject: [Rserpool] Services Provided By Reliable Server Pooling

From: [email-masked]
Date: Tue, 21 May 2002 13:44:37 -0500

Phil and Peter,

Here are my initial comments on draft-conrad-rserpool-service-00.txt section 3.2.

3.2 Example Scenario for RSerPool Nameservice Mode

I'm confused about the implementation of ASAP 03 and ENRP 03 for
applications using name service mode.

1) The document does not state if ASAP 03 as defined for the client needs to
be fully implemented or not.

----------------------- start of text -----------------------------------------------------------------------

2. Instead of using a DNS based service (e.g. the Unix library function gethostbyname()) to translate from a hostname to an IP address, the application will invoke an RSerPool service primitive "IPAddressOfPrimaryServer" that takes as input a pool handle, and returns the IP address of the primary server. The application then uses that IP address just as it would have used the IP address returned by the DNS in the previous scenario.

3. In nameservice mode, failure detection is application specific just as in the previous scenario. However, when failure is
detected on the primary server, instead of invoking DNS translation again on the hostname of a secondary server, the application invokes the service primitive "IPAddressOfNextServer", which has a dual meaning. First it indicates to the RSerPool layer the failure of the server returned by a previous "IPAddressOfPrimaryServer" or "IPAddressOfNextServer" call. Second, it provides the IP address of the next server that should be contacted, according to the best information available to the RSerPool layer at the present time.

----------------------------- end of text -------------------------------------------------------

We make it explicit that when messages are sent by the PU and received by
the PE, we don't use the ASAP layer. Specifically, there is no using ASAP
data.send.request and the data.response by the PU. The application does its
communication with the PE as already defined in its legacy application.
Again, this minimizes changes to the app. However, I'm assuming that pool
address caching still goes on in the PU as defined by ASAP. (Am I right
about all this? This is the kind of thing I want to know.)

2) Same comment for the PE. The document does not state if ASAP 03 as
defined for the PE (server) needs to be fully implemented or not. I was
especially confused by the 1. and 2. bullets saying effectively "just add this" to the server.

-------------------------------------- start of text -------------------------------------------------------

For pool element ("server") applications, two additions are required:

1. The server should invoke the REGISTER service primitive upon startup to add itself into the server pool using an appropriate pool handle. This also includes the address(es) protocol or mapping id, port (if required by the mapping), and pooling policy(s).

2. The server should invoke the DEREGISTER service primitive to remove itself from the server pool when shutting down.

------------------------------- end of text -----------------------------------------------------------------

Does this mean that you don't have to implement the full ASAP for the
server, but some subset that is vaguely defined here? For example, does the
server need to respond to an endpoint keep alive message?

Anyway, if this name service mode specifies some subset of ENRP 03 and ASAP
03 for name service mode, then that should be stated directly. Then the
subset should be specified in another document in a detailed manner. If all
of ENRP 03 and ASAP 03 are required to be implemented, then it should
clearly say this -- (the Right Thing in my opinion).

The goal is stated clearly -- that you want to minimize the changes to the
application in this mode and this is a highly desirable goal.

My opinion is that ASAP 03 and ENRP 03 should be fully implemented as
defined even if the application(s) only use name service mode. All we are
doing with name service mode is not taking advantage of the full underlying
power of ASAP and ENRP. The services exposed at the application layer is
different in each mode, thereby making it a smaller task to modify the
application for name service mode. The full power of ASAP and ENRP are there
if you want to tap into them later.

Does this make sense? Comments?

-- maureen

_______________________________________________
rserpool mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rserpool

---

From: Peter Lei <[email-masked]>
Date: Tue, 21 May 2002 14:09:49 -0500

Maureen,

My opinion is in line with yours and I agree, it should be clearer in the text:
I think that the full ASAP/ENRP protocols must be implemented at the PU and PE
and wasn't thinking at all of allowing/defining "ASAP-lite" or "ENRP-lite".
The nameservice mode simply provides the ULP/application a subset of the full
suite of services available with the full protocols.

We hope to be able to define the rserpool API such that the nameservice mode API
is (just) a subset of the full API-- regardless both modes would interact with
the same protocol stack.

--peter

---

From: Peter Lei <[email-masked]>
Date: Tue, 21 May 2002 14:09:49 -0500

Maureen,

--peter

---

From: Phillip Conrad <[email-masked]>
Date: Tue, 21 May 2002 18:31:22 -0400

Maureen:

I first want to indicate that I agree with and endorse Peter's point in his 

>suite of services available with the full protocols.

Exactly right.   Furthermore, the concept of "nameservice" versus 
"failover" mode (or, as I've called it in previous discussions 
"RSerPool-lite" vs. "full featured RSerPool" is really only about how a PU 
and a PE interact with each other.

>is (just) a subset of the full API-- regardless both modes would interact with

Exactly right.    There is no difference "on the wire" between the data 
exchange between a PE and a PU that uses nameservice mode, vs. one that 
uses failover mode.   As described right now, the ASAP protocol headers, 
etc. don't even play a role in data exchange... there is no message type 
for data exchange.  Instead, the data.send primitive is supposed to invoke 
the services of the appropriate transport protocol below, e.g. SCTP.

This is fine (to some extent), as long as SCTP is the only transport 
protocol that is going to be supported.  (Actually, there are still some 
unaddressed problems even here... for example, I'm having trouble 
understanding why, for example, the data send primitive has a flag for the 
"unordered" option of SCTP, but no way to indicate stream number...   my 
thinking is that this is just because we haven't gotten around to putting 
it in yet.)

But if you want to support other transport protocols as well, through the 
data.send primitive, then you have "some problems."  I think this is the 
key idea: the whole purpose of the "services" draft is to define clearly 
what "those problems" are, and then to propose a solution.

This may not be clear enough in the current text, and we can try to make it 
more clear.

In particular, those problems are:

(1) SCTP gives you a way to know if a connection failed or not (heartbeat 
mechanism) to let you know when to failover a connection, even if there is 
no active data on that connection.  (not true of TCP or UDP).

(2) SCTP provides reliable delivery and message orientation, so that if a 
message doesn't get delivered reliably to PE1, (a) we can find out, and (b) 
the message can be treated as an atomic unit (it has clearly defined 
boundaries) and it can be delivered instead to PE2 after failing 
over.   (And, we have "retrieval" in the API to assist with this.)  Not 
true of TCP; all we know is that a connection failed, and there is no way 
(at least not through std. APIs) to tell the upper layer what did or didn't 
get delivered.    And, UDP provides no reliable delivery at all.

So.. there are two ways to support alternate transports to SCTP, for the 
data exchange portion of the application.    One is to add the missing 
features into a shim layer that sits between the alternate transport 
protocol and the ASAP layer.  This is what we propose as a good option for 
TCP, and this is why we can support TCP in failover mode (full featured 
RSerPool).

See for example, a diagram of the stack for failover mode:

+--------------------------------------+
|   Application (Pool User)            |
+--------------------------------------+
|          ASAP   API                  |
| (PE selection)    (data exchange)    |
+-----------------+--------------------+
| ASAP protocol   |                    |
+-----------------+                    |
|  shim layer (null if SCTP underneath)|
+--------------------------------------+
  |                SCTP         or         TCP                          |
+--------------------------------------+

In failover mode, the shim layer provides for heartbeats,
message orientation, and retrieval when TCP is used instead of
SCTP.    (Note to Peter... I'll send you some initial thoughts about
the protocol for the shim layer, and maybe we can get a draft out
on that by the end of the week.)   For SCTP, the shim layer is
nothing more than a direct mapping from the RSerPool API
to the SCTP API. (*See footnote below.)

vs. for nameservice mode:

+--------------------------------------------+
|        Application (Pool User)             |
|                  (conn estab., data xchange|
|(PH->IP xlation)   failure detect, failover)|
+-----------------+--------------------------+
|   ASAP   API    |  Alternate transport     |
| (PE selection)  |   API (e.g. Sockets API  |
+-----------------+   for UDP, TCP, DCP,     |
| ASAP protocol   |   or any API for any     |
+-----------------+   other communications   |
|  shim layer     |   protocol(s)            |
|(null if SCTP    +--------------------------+
|underneath)      |   The alternate protocol |
+-----------------+   or protocols (UDP, TCP,|
|  SCTP  or TCP   |    DCP, or whatever.     |
+-------------------------------------------+

The beauty of this scheme is that it allows RSerPool to be used
for any client/server application regardless of the protocol (or even
the combination of protocols!)  used for the data exchange portion.
The catch for the application designer is that they have to provide,
as part of the application, a lot of the functions provided by RSerPool
when used in failover mode.    Of course, if someone wants to
provide a shim layer for UDP, DCP that would be possible (though
PR-SCTP might be a better alternative.)

(*Footnote: I envision, by the way, extending the data.send primitive to
include the full set of features that would be available for sending
data using SCTP, including stream numbers, etc.  It should be possible
to move from using SCTP without RSerPool to using SCTP with RSerPool
without having to rethink your application too much.    Does anyone see
a  conflicts  with some other part of the architecture here?)

>be fully implemented or not.

We'll clarify that ASAP (the *protocol*) MUST be fully implemented for 
nameservice mode to work properly.  The set of primitives offered might end 
up being a subset, however.  We'll need to clarify which set of primitives 
belong in the subset vs. the superset.   Essentially, we are probably doing 
a re-write of the service primitive part of the ASAP draft, and if this 
concept is embraced, what we put in this draft might need to eventually get 
folded in there, or vice-versa... if the service draft lives on as a 
separate document, maybe we move the "service" definition of ASAP into the 
service draft.

>----------------------- start of text 
>-----------------------------------------------------------------------
>
>2. Instead of using a DNS based service (e.g. the Unix library function 
>gethostbyname()) to translate from a hostname to an IP address, the 
>application will invoke an RSerPool service primitive 
>"IPAddressOfPrimaryServer" that takes as input a pool handle, and returns 
>the IP address of the primary server. The application then uses that IP 
>address just as it would have used the IP address returned by the DNS in 
>the previous scenario.
>
>3. In nameservice mode, failure detection is application specific just as 
>in the previous scenario. However, when failure is
>detected on the primary server, instead of invoking DNS translation again 
>on the hostname of a secondary server, the application invokes the service 
>primitive "IPAddressOfNextServer", which has a dual meaning. First it 
>indicates to the RSerPool layer the failure of the server returned by a 
>previous "IPAddressOfPrimaryServer" or "IPAddressOfNextServer" call. 
>Second, it provides the IP address of the next server that should be 
>contacted, according to the best information available to the RSerPool 
>layer at the present time.
>
>----------------------------- end of text 
>-------------------------------------------------------
>

Yes, you are correct.. and it is a good catch that we should make all of 
this more clear and explicit.

>especially confused by the 1. and 2. bullets saying effectively "just add 

Right... I'll revisit that text and try to make it more clear.  In general, 
I need to do a complete pass through all the related RSerPool drafts, but 
especially the ASAP and Architecture drafts to try to find all the issues 
such as the ones you raise here.

>-------------------------------------- start of text 
>-------------------------------------------------------
>
>1. The server should invoke the REGISTER service primitive upon startup to 
>add itself into the server pool using an appropriate pool handle. This 
>also includes the address(es) protocol or mapping id, port (if required by 
>the mapping), and pooling policy(s).
>
>2. The server should invoke the DEREGISTER service primitive to remove 
>itself from the server pool when shutting down.
>
>------------------------------- end of text 
>-----------------------------------------------------------------
>

Hmmm... no we didn't mean to imply that, because I think we were assuming 
that the full ASAP implementation is still sitting under the application, 
and IT (not the application per se) is responding to endpoint keep alive 
messages.   According to ASAP draft 03, section 2.2.7:

>This message is sent to a PE by the ENRP server has a "health"
>     check.

(I think that should be "as a" not "has a"... anyway....)  So the "server" 
will respond to these, but here I think that by "server", you mean the 
"box".   But it is actually the ASAP entity on the PE machine that is 
responding to the endpoint keep alive.

And as we said, regardless of whether we are in nameservice or failover 
mode, the PE *MUST* support SCTP (no other protocol) because the ENRP 
servers are only going to speak SCTP.  So this particular protocol element 
of ASAP will always be sent using SCTP.. and in some sense is out of scope 
for the services document.

If we are more precise about our language, actually, the "application 
server", i.e. the Pool Element, is not responding to these messages at 
all.  In the architecture document, look at the diagrams in sections 2.2.3, 
2.2.4, and 2.2.5.   The PE is defined as the entity that SITS ON TOP OF the 
ASAP entity.  It is in fact the ASAP entity to which the keep alive is 
sent, not the PE, per se.  So even the current ASAP draft is a bit 
imprecise in this sense.

>clearly say this -- (the Right Thing in my opinion).

Agreed :-)

>Does this make sense? Comments?

Yes, absolutely.   Thanks for reading and providing comment... I'm 
energized now to do some more writing on this and try to tighten up some of 
the remaining open issues.   It will also probably get other people on the 
list to perhaps read the drafts and express some opinions :-)

Thanks,
Phill

---

From: Phillip Conrad <[email-masked]>
Date: Tue, 21 May 2002 18:31:22 -0400

Maureen:

Agreed :-)

Thanks,
Phill