Subject: [Rserpool] RSERPOOL TFTP: an application context transfer example

From: Erik Guttman <[email-masked]>
Date: Wed, 5 Sep 2001 15:43:47 +0200 (MET DST)

I think it would be helpful to include real examples in the
architecture doc to illustrate the issues with supporting real
existing applications with RSERPOOL.

Is this worth writing up in an internet draft form?  If so,
I will need a coauthor, since I'm rapidly running out of time
to work on this.

----------------------------------------------------------------------

In the case of what I call a 'stateless transaction protocol'
this is easy.  For example, DNS can be used transparently over
RSERPOOL.

    DNS                     DNS          DNS
   RESOLVER                SERVER       SERVER
   . . . . .               . . . .      . . . .
     PU                     PE1          PE2
    ====        RQST1      =====        =====
  RSERPOOL -------------> RSERPOOL     RSERPOOL
           <------------
                RPLY1

                RQST2
           -------------------------->
           <-------------------------  
                RPLY2

                RQST3
           ------------->
           <--------------------------
                RPLY3

RQST1/RPLY1 is independent of RQST2/RPLY2.  The second exchange
would work equally well if the first exchange did not occur.

!!! Note that RQST3 sent to PE1 would require state sharing if RPLY3 !!!
!!! were to be retrieved from PE2.  I do not believe that is what    !!!
!!! RSERPOOL is intended to do, though, right?                       !!!

The example above would also work for HTTP with chunking based
requests, I believe.  That is, RSERPOOL could switch between 
HTTP servers as long as the individual requests were self-identifying
as far as which data and which offset into the data to be retrieved.
(This also assumes the HTTP servers in the pool offer the same

A much harder example to work out is one where there is application
layer state which has to be shared among PEs.  To make this a tractable
problem (I do have a day job :-) I'm choosing TFTP [RFC 1350].  

Like most Internet application suites, TFTP's notion of a session 
is tightly bound to the underlying transport - in this case UDP.  
UDP port numbers as transfer identifiers (TIDs).  

The first challenge of mapping TFTP to RSERPOOL is breaking the
connection between application and transport layer.  The TFTP
client need to be changed so that they use a session layer API
instead of the transport API.

I want to work through this carefully, so I'll use 'realistic' pseudocode.

---------------------------------------------------------------------------
---------------------------------------------------------------------------
Client side - traditional
{
  /* initialize */
  s = socket(AF_INET,SOCK_DGRAM,0);
  first = 1;
  done = 0;
  sockaddr_in = {server location};
  /* initialize buffers and lengths - not shown */

  while (!done) 
  {
    /* get_data does all work: keeps state machine & decides when done. */
    get_data(&outdata,&outlen,indata,inlen,&sockaddr_in_r,&done);
    sendto(s,outdata,outlen,0,(struct sockaddr*)&sockaddr_in,sz);
    inlen = recvfrom(s,indata,max,0,(struct sockaddr*)&sockaddr_in_r,&sz);

    /* change destination to reflect transport association to server */
    if (first) 
    {
      s = change_dest(sockaddr_in_r);
      first = 0;
    }
  }
  close(s);
}

---------------------------------------------------------------------------
Client side - RSERPOOL based, much simpler!

{
  s = session_create(pool_info);  /* a handle, name or description */
  done = 0;
  /* initialize buffers and lengths */
  while (!done) 
  {
    /* get_data does all work: keeps state machine & decides when done. */
    get_data(&outdata,&outlen,indata,inlen,&done);
    session_send(s,outdata,outlen);
    inlen = session_recv(s,indata,max);
  }
}

---------------------------------------------------------------------------
---------------------------------------------------------------------------
Server-side state machine and stored state

From the point of view of the server, the following exchanges are
possible.

(1)

  C                          S
  o--Read Request (RRQ)-->
             <--Data (DATA)--o
  o--Acknowledgment (ACK)-->
            ...
             <--Data (DATA)--o
  o--Acknowledgment (ACK)-->
  
(2)
  o--Write Request (WRQ)-->
   <--Acknowledgement (ACK)--o
  o--Data (DATA)-->
   <--Acknowledgement (ACK)--o
            ...   
  o--Data (DATA)-->
   <--Acknowledgement (ACK)--o

Notes:
 - Each request has a block id.  It is associated with a reply.
   Pairs are WRQ-ACK, DATA-ACK... or RRQ-DATA, ACK-DATA.
 - If the reply does not arrive, the request is sent sent again.
   (retransmission issues ignored in the code above, but have to
   be considered in the stateful app context transfer).
 - In any case, an ERROR may be returned (not shown).
 - The final ACK is optional - since the last DATA packet with
   a length < 512 indicates that its the final one.  This DATA
   has to be held for a while, since the *preceding* ACK (or request)
   may be sent again.  This indicates that the (final) DATA was not 
   received.  Thus it is 'polite' to send the final ACK.

Server session state includes:
 - session identifier (UDP port number in TFTP, something else in
   RSERPOOL enhanced TFTP)
 - Last block id
 - Last data block to transfer
 - File to read must be available at the server.
 - File that is being written. (This is kind of weird - a WRQ to
   a server pool will have to be written to ALL servers in the pool.
   That's the only way, for example, that it can be subsequently
   read by the client using a RRQ.)
 - Mode of request (netascii, octet or mail).  Determines the
   behavior of the reply.
 - Which operation is being performed (ie. on which file, etc.)

This server state can be derived from 
 * the initial WRQ or RRQ from the client
 * the client's address and udp port (session id in the RSERPOOL case)
 * the server's udp port number (session id in the RSERPOOL case)
 * the last datagram sent to the client (which may have to be sent again,
   but can be discarded once it has been acknowledged with the next ACK
   or DATA in block-id sequence)

We need session ids.  Two possibilities:
1) We enhance TFTP with session IDs - for example prepend a 16 byte
   unique id SID preamble on the TFTP protocol (as a 'session header').
2) We assume that RSERPOOL includes the maintenance and transport 
   of session ids.  For example, when session_create() is called on
   the client, this occurs implicitly.  The, the server can call
   session_get_sid(s) for any session handle and find out what 
   session it is.

Either way, we need the SID for the PEs to associate incoming
requests with various outstanding sessions.

---------------------------------------------------------------------------
Server side - traditional

handler(void *p) 
{
  /* initialize buffers and lengths */
  param = (paramtype*) p;
  done = 0;
  while (!done) 
  {
    /* get_data does all work: keeps state machine & decides when done. */
    get_data(&outdata,&outlen,p,&done);
    sendto(p->s_session,outdata,outlen,0,
           (struct sockaddr*)p->insin,sizeof(struct sockaddr_in));
    recvfrom(p->s_session,p->indata,p->max,0,
             (struct sockaddr*)p->insin,sizeof(struct sockaddr_in));
  }
  close(p->s_session);
  freeparam(param);
}

listener()
{
  /* initialize buffers and lengths */
  s = socket(AF_INET,SOCK_DGRAM,0);
  struct sockaddr_in mysin = {AF_INET, 69, INADDR_ANY};
  struct sockaddr_in anysin = {AF_INET, 0, INADDR_ANY};
  bind(s,mysin,sizeof mysin);
  while(1) /* forever */
  {
    inlen = recvfrom(s,indata,max,0,(struct sockaddr*)&insin,&sz);
    /* The new session will be assigned an ephemeral port (TID).
       Use this new session in future communication with the client. */
    s_session = socket(AF_INET,SOCK_DGRAM,0);
    bind(s_session, anysin, sizeof anysin);
    param = packparam(indata,inlen,max,insin,s_session);
    pthread_create(&thr_id, handler, (void*)param);
  }
}

The reason I chose a threaded implementation is (a) it shows
how the session state is distict which will be useful later, 
(b) cleaner exposition.

---------------------------------------------------------------------------
Server side - RSERPOOL based, much more complicated!

Assume we have primitives

  session_send_all(s,msg,len);             sends msg to all PEs in pool
  session_barrier(s,msg,max,callback);     blocks till received from all 
                                           PEs in pool: callback evoked
                                           each time data received if
                                           defined.
                                        
state; /* global.  encapsulates all distribute shared TFTP state */

handler()
{

  inlen = session_recv(s,indata,inlen);
  change = get_data(indata,inlen,outdata,&outlen,max,sid);
  /* In case of a state change, that is, the client asks for the
     next piece of data, make sure all other PEs are updated with
     the correct state before actually responding to the client 
   */
  if (change)
  {
    update_msg_create(updata,uplen,sid);
    state_lock();
    session_send_all(handle.h,updata,uplen);
    session_barrier(handle.h,respdata,max,NULL);
    state_unlock();
  }
  /* send response to app */
  session_send(s,outdata,outlen);  
}

listener()
{
  /* initialize buffers and lengths */
  pTFTP  = get_pool_handle(description_or_name);
  state.pSTATE = get_pool_handle(description_or_name);
  s = session_join(pTFTP);
  pthread_create(&thr_id,session_maint,(void*)NULL);

  while(1)
  {
    inlen = session_recv(s,indata,max);
    sid   = session_get_sid(s);  /* or = get_sid(indata,inlen); */
    if (!state_exists(sid))
    {
      sid = state_init(indata, indata, inlen, max);
    }
    process(s,sid,indata,inlen,max);
  }
}

/* - - - - - - - session state maintenance code below - - - - - - */

state_update_handler(); /* not shown:  Add record for latest 
                           server-side data block */
state_update_init();    /* not shown:  Set up state associated with
                           SID, mode, request type, initial data block... */

session_maint(void *)
{
  state.h = session_join(state.pSTATE); /* join the pool for listening */
  while(1) 
  {
    inlen = session_recv(state.h,indata,max);
    if (lock_msg(indata,inlen))
    {
      lock(state.mutex);
    }
    else if (unlock_msg(indata,inlen)) 
    {
      unlock(state.mutex);
    }
    else if (update_msg(indata,inlen)) 
    {
      update_state_handler(indata,inlen);
    }
    else if (init_msg(indata,inlen))
    {
      init_state_handler(indata,inlen); 
    }
  }
}

state_lock()
{
  create_msg(outdata,&outlen,max,LOCK,sid);
  session_send_all(state.h,outdata,outlen);

  /* NOTE deadlock will result from this if >1 request a lock
     at the same time.    

     We need a way to avoid this, further complicating the 
     distributed state synchronization problem.  I won't go here - 
     but essentially, the protocol for locking the state would associate 
     an ordered ID for each PE.  If a PE with a higher ID wants the lock,
     everyone knows the lower ID loses out and has to wait.  */

  session_barrier(state.h,indata,max,NULL);
}

state_unlock()
{
  create_msg(outdata,&outlen,max, UNLOCK,sid);
  session_send_all(state.h,outdata,outlen);
  /* see note for state_lock */
  session_barrier(state.h,outdata,outlen,NULL);
}

state_exists(pSTATE,sid) 
{ 
  state_lock();
  if (contains(state,sid) ret = 1;
  else ret = 0;
  state_unlock();
  return ret; 
  /* OK, OK, there's a race condition here if >1 PEs try to init
     at once.  The only way to avoid this is to synchronize the
     PEs to prevent this race, or to detect races and resolve them.
     In both cases this is complicated.  The way I will solve this
     is by making state_lock and state_unlock a distributed operation.  
     Yuck! */
}

state_init(sid,indata,inlen,max) 
{
  create_init_msg(outdata,&outlen,max,sid);
  state_lock();
  /* update all 
  session_send_all(state.h,outdata,outlen);
  session_barrier(state.h,indata,max,NULL);
  state_unlock();
}

---------------------------------------------------------------------------
---------------------------------------------------------------------------
Lessons learned:

 - Application state synchronization, even in a trivial case
   is non-trivial, because
 - it is a distributed atomic transaction, requiring distributed locks

Note that my algorithm is 'eager':  It requires state distribution
before server-client operation.  This is not very efficient, though
it guarantees that if C starts talking to S2 instead of S1 it will
be able to - immediately.

    C        S1     S2     S3
      \_____
       rqst    \__________
                 sync    
                 ...
                __________/       The sync operations between
                 sync             S1-S2 and S1-S3 can happen
                                  in parrallel.
               \____
                 sync
                 ...
                ____/

      ______/   
       reply

Another possibility would be to be 'lazy'.  This would not require
state updates between servers on every interaction, only when the
client shifts from one server to another.  This presents certain
problems:

    C        S1   S2    S3
     \_______
       rqst  
      _______/
       rply
     \___________
       rqst
              ___/
              sync
              ...
             \____
              sync
       __________/
        rply
                      
 - The client C has to be aware of which server S it is communicating
   with.  When C communicates with a new server S', S' can go to S and
   get synchronization state.

 - Is it the client app which keeps track of and S and S'?  This burdens
   each application with state context transfer signalling.  Or is it the 
   session layer?  In this case the session layer could signal S' to get
   a state transfer from S and greatly simplify the app server (in this
   case the TFTP server).

 - Robustness is lost.  If S1 dies (likely, if C wants to use S2), there
   is a great chance that S2 will not be able to get state from S1.

----------------------------------------------------------------------

Erik

_______________________________________________
rserpool mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/rserpool