Subject: [AAA-WG]: Proposed resolution of issue #173

From: Bernard Aboba <[email-masked]>
Date: Mon, 24 Sep 2001 17:06:58 -0700 (PDT)

This issue related to differences in the failover algorithm between
diameter-07 and transport-04. 

The proposed solution is to include failover text in only one place, the
transport draft. Diameter base will reference that text. 

Of course, there is then the issue of what the text should be. A proposal
for the next text of section 3.2 of transport -05 is included below. This
proposal contains a single timer, Tw. 

Please send comments to the list. 

------------------------------------------------------------------

3.2  Application layer watchdog

In  order  to  enable  AAA  implementations  to more quickly
detect transport and application-layer failures, AAA  proto
cols MUST support an application layer watchdog message.

The application layer watchdog message enables failover from
a server that has failed either because it is unreachable or
because its applications functions have failed. This is dis
tinct from the purpose of the SCTP heartbeat,  which  is  to
enable  failover  between interfaces. The SCTP heartbeat may
enable a failover to another path to reach the same  server,
but does not adress the situation where the server system or
the application service has failed.  Therefore  both  mecha
nisms MAY be used together.

The  watchdog  is  used  in  order to enable a AAA client or
agent to determine when to resend on another connection.  It
operates  on all open connections and is used to suspend and
eventually close connections that are experiencing difficul
ties. The watchdog is also used to re-open and validate con
nections that have returned to health.  The watchdog may  be
utilized  either  within primary/secondary or load balancing
configurations.  However, it is not intended  as  a  cluster
heartbeat mechanism comparable to that proposed in [31].

3.2.1.  Algorithm overview

The  watchdog behavior is controlled by an algorithm defined
in this section. Implementations SHOULD implement this 
algorithm, which operates as follows:

[1]  Watchdog behavior is controlled by a single timer (Tw).
     The default value of Tw is 30 seconds. This value was
     selected because it minimizes the probability that
     failover will be initiated due to a routing flap,
     as noted in [Paxson]. 

     While Tw MAY be set as low as 6 seconds (not including
     jitter), it MUST NOT set lower than this. Note that
     setting such a low value of the watchdog timer is
     likely to result in an increased probability of
     duplicates, as well as an increase in spurious
     failover and failback attempts. 

     In order to avoid synchronization behaviors that can 
     occur with fixed timers among distributed systems, each
     time the watchdog interval is calculated with a jitter
     by using the Tw value and randomly adding a value drawn
     between -2 and 2 seconds. Alternative calculations to 
     create jitter MAY be used. These MUST be pseudo-random
     and not cyclic. 

[2]  When a response is received, Tw is reset.  Receiving  a
     watchdog  from  a  peer  constitutes  activity,  and Tw
     should be reset.  On sending a  message,  if  the  send
     queue is empty, then Tw is reset. If the watchdog timer
     expires and the send queue is empty,  then  a  watchdog
     packet  is sent. Watchdog packets are not retransmitted
     by the AAA protocol, since AAA protocols run over reli
     able  transports  that  will handle all retransmissions
     internally.

     If the watchdog timer expires and the send queue is not
     empty,  then  failover is initiated. The AAA client MAY
     resend the request to an alternate server, reusing  the
     end-to-end  identifier so as to permit duplicate detec
     tion. However, the client MUST NOT  close  the  primary
     connection  until  the  primary's  watchdog  timer  has
     expired twice without a response (note that the  watch
     dog  is not sent a second time, however). Once the pri
     mary connection has  failed,  subsequent  requests  are
     sent  to  the alternate server until the watchdog timer
     on the primary connection is reset.

     Suspension of the primary connection prevents  flapping
     between  primary and alternate connections, and ensures
     that failover behavior remains consistent.  The  appli
     cation  may not receive a response to the watchdog mes
     sage due to a connectivity problem,  in  which  case  a
     transport layer ACK will not have been received, or the
     lack of response may be due to an application  problem.
     Without  transport layer visibility, the application is
     unable to tell the difference, and must behave  conser
     vatively.

     In  situations where no transport layer ACK is received
     on the primary connection after  multiple  re-transmis
     sions, the RTO will be exponentially backed off. Due to
     Karn's algorithm as implemented SCTP and TCP,  the  RTO
     estimator  will  not  be  reset  until  another  ACK is
     received in response to a  non-re-transmitted  request.
     Thus,  in  cases where the problem occurs at the trans
     port layer, after the client fails over to  the  alter
     nate  server,  the  RTO of the primary will remain at a
     high value unless an ACK is  received  on  the  primary
     connection.

     In  the  case where the problem occurs at the transport
     layer, subsequent requests sent on the primary  connec
     tion  will  not  receive the same service as was origi
     nally provided. For example, instead of failover occur
     ing after 3 retransmissions, failover might occur with
     out even a single retransmission if RTO has been suffi
     ciently  backed off. Of course, if the lack of a watch
     dog response was due to an application  layer  problem,
     then  RTO will not have been backed off. However, with
     out transport layer visibility, there is no way for the
     application to know this.

     Suspending  use  of  the  primary  connection  until  a
     response is received to a watchdog  message  guarantees
     that the RTO timer will have been reset before the pri
     mary connection is reused. If no response  is  received
     after  the  second Tw expiration, then the primary con
     nection is closed and so the suspension becomes  perma
     nent.

[3]  After the the expiration of two watchdog timers without
     a response, the AAA client  SHOULD  cause  a  transport
     reset or close to be done on the connection.  While the
     connection is in the closed state, the AAA client  MUST
     NOT  attempt  to  send further watchdog messages on the
     connection. However, after the  connection  is  closed,
     the AAA client continues to periodically attempt to re-
     open the connection.  The AAA client  SHOULD  wait  for
     the transport layer to report connection failure before
     attempting again, but MAY choose  to  bound  this  wait
     time by the watchdog interval, Tw. If the connection is
     successfully opened, then the watchdog message is sent.
     Once   three  watchdog  messages  have  been  sent  and
     responded to, the connection is  returned  to  service,
     and  transactions  are once again sent over it. Connec
     tion validation via receipt of  multiple  watchdogs  is
     not  required when a connection is initially brought up
     -- in this case, the connection can immediately be  put
     into service.

     When  using SCTP as a transport, it is not necessary to
     disable SCTP's transport-layer heartbeats.  However  If
     AAA  implementations  have  access  to SCTP's heartbeat
     parameters, they MAY chose to ensure that SCTP's heart
     beat  interval is longer than the AAA protocol's watch
     dog interval, Tw.  This will ensure both that alternate
     paths  are still probed by SCTP, while the primary path
     has a minumum of heartbeat redundancy.

In the event that a transport failure  is  detected  with  a
peer, it is necessary for all pending request messages to be
forwarded to an alternate agent, if possible. This  is  com
monly referred to as failover.

In  order for a AAA client or agent to perform failover pro
cedures, it is necessary to maintain a pending message queue
for  a  given  peer. When an answer message is received, the
corresponding request is removed from the queue. The Hop-by-
Hop  Identifier  field  MAY be used to match the answer with
the queued request.

When a transport failure is detected, all  messages  in  the
queue are sent to an alternate agent, if possible.  Multiple
identical requests or answers MAY be received as a result of
a  failover. The combination of an end-to-end identifier and
the origin host MUST be used to identify duplicate messages.

3.2.2.  Detailed algorithm description

In this section, we will refer to a memory control structure
that contains all information regarding a specific peer as a
Peer Control Block, or PCB.

For  the  purposes  of  illustrating the algorithm, each PCB
contains the following fields:

      Status  - This field represents the level of confidence in the
                algorithm. The following values are defined:

                  OKAY:       The connection is up
                  SUSPECT:    Failover has been initiated on the connection.
                  DOWN:       Connection has been closed.
                  REOPEN:     Attempting to reopen a connection

      Pending - This boolean field is set to TRUE when there are no
                outstanding unanswered requests.

T is the watchdog timer, measured in seconds. Every  second,
T  is  decremented.  When  it reaches 0, the OnTimerElapsed
event (see below) is invoked.

Pseudo-code for the algorithm is as follows:

/*

Here is the Failover state machine:

States:
  OKAY:       The connection is up
  SUSPECT:    Failover has been initiated on the connection.
  DOWN:       Connection has been closed.
  REOPEN:     Attempting to reopen a closed connection
  Pending:    Set to TRUE if there are outstanding unanswered requests
  T:          Watchdog timer value

STATE         Event                New State         Actions
=====         =====                =========         =======
OKAY          Send &               OKAY              SetWatchDog()
              !Pending
OKAY          Send &
              Pending              OKAY
SUSPECT       Send                 ERROR             Error()
DOWN          Send                 ERROR             Error()
OKAY          Receive              OKAY              SetWatchDog()
SUSPECT       Receive              OKAY              Failback()
                                                     SetWatchDog()
DOWN          Receive              DOWN              Throwaway()
OKAY          Timer expires &      OKAY              SendWatchdog()
              !Pending
OKAY          Timer expires &      SUSPECT           FailOver()
              Pending                                SetWatchdog()
SUSPECT       Timer expires        DOWN              Close Connection
                                                     SetWatchdog()
DOWN          Timer expires        REOPEN            AttemptOpen()
                                                     SetWatchdog()
REOPEN        Timer expires        REOPEN            SetWatchdog()
REOPEN        Connection up        SUSPECT           SendWatchdog()
OKAY          Connection down      DOWN              SetWatchdog()
SUSPECT       Connection down      DOWN              SetWatchdog()
REOPEN        Connection down      DOWN              SetWatchdog()
DOWN          Connection down      ERROR             Error()
*/

/* SetWatchdog() is called in order to set the Watchdog
   timer, measured in seconds. The default value of
   TWINIT is 30 seconds. A random value between
   -2 and 2 seconds is added to the timer to provide jitter.
*/

SetWatchdog()
{
    return (TWINIT -2.0 + 4.0 * random()) ;
}

/*
 OnReceive() is called whenever a message
 is received from the peer. This message MAY
 be a request or an answer, and can include
 DWR and DWA messages.
*/
OnReceive(pcb)
{
   switch (pcb->Status){
   case OKAY:
        T = SetWatchdog();
        break;
   case SUSPECT:
        pcb->Status = OKAY;
        T = SetWatchdog();
        Failback();
        break;
   case REOPEN:
        pcb->Status = SUSPECT;
        T = SetWatchdog();
        break;
   case DOWN:
        Throwaway(received packet);
        break;
   default:
        Error("Shouldn't be here!");
        break;
   }
}

/*
OnSendRequest() is called after a request is sent to
the peer.
[Issue: Should this be called whenever any message is
sent to the peer?]
*/
OnSendRequest(pcb)
{
   switch (pcb->Status){
   case OKAY:
        if (!Pending) T = SetWatchdog();
        break;
   default:
        error("Shouldn't be here!");
        break;
   }
}

/*
OnTimerElapsed() is called whenever T reaches zero (0).
*/
OnTimerElapsed(pcb)
{
    switch (pcb->status){
       case OKAY:
          if (!Pending) {
             SendWatchdog(pbc);
             break;
             }
          pcb->status = SUSPECT;
          FailOver(pcb);
          break ;
       case SUSPECT:
          pcb->status = DOWN;
          CloseConnection(pcb);
          OpenAttempt = FALSE;
          T = SetWatchdog();
          break;
       case DOWN:
          if (!OpenAttempt) {
             OpenAttempt = TRUE;
             AttemptOpen(pcb);
             break; 
          }
          break;
       default:
          error("Shouldn't be here!);
          break;
       }
}

/*
OnConnectionUp() is called whenever a connection comes up
*/
OnConnectionUp(pcb)
{
    switch (pcb->status){
       case DOWN
          pcb->status = REOPEN;
          SendWatchdog();
          break;
       default:
          error("Shouldn't be here!);
          break;
       }
}

/*
OnConnectionDown() is called whenever a connection goes down
*/
OnConnectionDown(pcb)
{
    switch (pcb->status){
       case OK:
          pcb->status = DOWN;
          T=SetWatchdog();
          break;
       case SUSPECT:
          pcb->status = DOWN;
          T=SetWatchdog();
          break;
       case REOPEN:
          pcb->status = DOWN;
          T=SetWatchdog();
          break;
       default:
          error("Shouldn't be here!);
          break;
       }
}

---

From: "Fredrik Johansson" <[email-masked]>
Date: Tue, 25 Sep 2001 22:17:15 +0200

do we really need a new draft that we'll be dependent on? We already have
three drafts as it is today. Why not make the transport draft an
informational draft and keep all necessary information about diameter
specific transport mechanisms in the base draft

my 2c
/Fredrik

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>Bernard Aboba
>Sent: den 25 september 2001 02:07
>To: [email-masked]
>Subject: [AAA-WG]: Proposed resolution of issue #173
>
>

---

From: Tony Johansson <[email-masked]>
Date: Tue, 25 Sep 2001 14:57:08 -0700

Bernard,

So, doing this means that the Diameter Base protocol, the Diameter 
NASREQ application, the Diameter MIPv4 application, the Diameter 
CMS-Security application and the Transport must all be submitted as a 
set to final call. Right? If so, do we really want to spread 
dependencies over yet another draft?

I'm very concerned that this could add on even more delay before we get 
Diameter approved, but I may have misunderstood the intention with this 
proposal.

Regards,

/Tony

Bernard Aboba wrote:

>From: Bernard Aboba <[email-masked]>
>To: [email-masked]
>Message-ID: <[email-masked]>
>MIME-Version: 1.0
>Content-Type: TEXT/PLAIN; charset=X-UNKNOWN
>Content-Transfer-Encoding: QUOTED-PRINTABLE
>Sender: [email-masked]
>Precedence: bulk
>
>diameter-07 and transport-04.=20
>
>transport draft. Diameter base will reference that text.=20
>
>proposal contains a single timer, Tw.=20
>
>Please send comments to the list.=20
>
>detect transport and application-layer failures, AAA  proto=AD
>because its applications functions have failed. This is dis=AD
>the application service has failed.  Therefore  both  mecha=AD
>eventually close connections that are experiencing difficul=AD
>ties. The watchdog is also used to re-open and validate con=AD
>in this section. Implementations SHOULD implement this=20
>     as noted in [Paxson].=20
>     failover and failback attempts.=20
>
>     In order to avoid synchronization behaviors that can=20
>     between -2 and 2 seconds. Alternative calculations to=20
>     and not cyclic.=20
>     by the AAA protocol, since AAA protocols run over reli=AD
>     end-to-end  identifier so as to permit duplicate detec=AD
>     expired twice without a response (note that the  watch=AD
>     dog  is not sent a second time, however). Once the pri=AD
>     that failover behavior remains consistent.  The  appli=AD
>     cation  may not receive a response to the watchdog mes=AD
>     unable to tell the difference, and must behave  conser=AD
>     vatively.
>
>     on the primary connection after  multiple  re-transmis=AD
>     Thus,  in  cases where the problem occurs at the trans=AD
>     port layer, after the client fails over to  the  alter=AD
>     layer, subsequent requests sent on the primary  connec=AD
>     tion  will  not  receive the same service as was origi=AD
>     nally provided. For example, instead of failover occur=AD
>     ing after 3 retransmissions, failover might occur with=AD
>     out even a single retransmission if RTO has been suffi=AD
>     ciently  backed off. Of course, if the lack of a watch=AD
>     then  RTO will not have been backed off. However, with=AD
>     that the RTO timer will have been reset before the pri=AD
>     after  the  second Tw expiration, then the primary con=AD
>     nection is closed and so the suspension becomes  perma=AD
>     nent.
>
>
>     and  transactions  are once again sent over it. Connec=AD
>     parameters, they MAY chose to ensure that SCTP's heart=AD
>     beat  interval is longer than the AAA protocol's watch=AD
>forwarded to an alternate agent, if possible. This  is  com=AD
>In  order for a AAA client or agent to perform failover pro=AD
>                  SUSPECT:    Failover has been initiated on the connection=
>=2E
>=3D=3D=3D=3D=3D         =3D=3D=3D=3D=3D                =3D=3D=3D=3D=3D=3D=
>=3D=3D=3D         =3D=3D=3D=3D=3D=3D=3D
>        T =3D SetWatchdog();
>        pcb->Status =3D OKAY;
>        T =3D SetWatchdog();
>        pcb->Status =3D SUSPECT;
>        T =3D SetWatchdog();
>        if (!Pending) T =3D SetWatchdog();
>        break;
>   default:
>          pcb->status =3D SUSPECT;
>          pcb->status =3D DOWN;
>          OpenAttempt =3D FALSE;
>          T =3D SetWatchdog();
>             OpenAttempt =3D TRUE;
>             break;=20
>          pcb->status =3D REOPEN;
>          pcb->status =3D DOWN;
>          T=3DSetWatchdog();
>          pcb->status =3D DOWN;
>          T=3DSetWatchdog();
>          pcb->status =3D DOWN;
>          T=3DSetWatchdog();

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 25 Sep 2001 16:35:32 -0700 (PDT)

> set to final call. 

The IESG has already indicated that this was required anyway. 
All of these drafts will go to the IESG as a unit. 

In any case, I'd worry less about where the algorithm ends up (this can be
changed with a cut and paste), than whether it makes sense.