Subject: RE: [Megaco] MGC-MG dialogue failure detection

From: Carl Rutter <[email-masked]>
Date: Wed, 10 Jul 2002 09:25:05 -0400

This is a good discussion,
We have been debating some of these
MEGACO issues for large Media Gateways
internally for awhile.

We have a very high density MG and some of the MEGACO
issues I run into are related to the fact that we have many,many
terminations.  ie) The Huge Reply thread for Wildcard Audits etc
                   currently under discussion
20,000+ terminations 
with 100 Calls Per Second not uncommon.

Christian brings up a valid point,
Maybe the thinking on this should be 
If a failover time of less than  (Notify Reply Timeout + retries) is desired
then the MGC should implement hot-swap.

Carl

-----Original Message-----
From: Christian Groves [mailto:[email-masked]]
Sent: Tuesday, July 09, 2002 10:16 PM
To: [email-masked]
Subject: Re: [Megaco] MGC-MG dialogue failure detection

Hello,

This has been discussed on multiple occasions. As Kevin points out having a
secondary take over causes control conflicts and opens up the chance that a
secondary MGC fraudulently takes over control.

The current handoff mechanism doesn't rely on shared state as the secondary
MGC
can always audit the MG to build state information. If you are not
implementing
hot swap on the MGC then I think this software handover is not practical if
the
MGC goes down unannounced. The call control messages will have to be
rerouted to
the new MGC which takes time. If the MGC/MG interface has sufficient load
the MG
will detect failure in a short period of time. Therefore any new call
attempts
are likely to be minimal.

Perhaps what we need are some real (or likely) figures on number of calls
attempts and hour, number of messages sent across the MGC/MG interface and
likely delay across the MGC/MG interface to see what the magnitude of the
problem actually is.

Regards, Christian

Aleksandr Ryabin wrote:
> 
> Hi,
> 
>         I agree that it is useful, I mean leaving up to MGCs to decide.
> More over I do not think that secondary MGC taking over will be a problem
from
> 
> synchronization point of view with primary MGC. In order to correctly to
take
> over
> even with a regular handoff initiated by primary MGC, both MGCs must have
the
> same state information, thus they must have some kind of synchronization
> mechanism in the first place to perform a regular handoff. Thus they
should be
> 
> always aware of who is current active MGC.
> 
> Thanks
>         Aleks
> 
> At 04:39 PM 7/9/2002, Carl Rutter wrote:
> 
> > Yes,
> >
> > The down side of this approach would put the burden on the Secondary
MGCs to
> > make sure
> > IF they are doing a Handoff they are only taking over for a downed
Primary.
> > But putting the decision making up in the MGCs, I believe, would be a
useful
> > thing.
> > I think the Inactivity mechanism would still stand.  This would be
another
> > way
> > of handling the Primary unexpectedly going down.
> >
> >
> > Carl
> >
> >
> >      -----Original Message-----
> >      From: Kevin Boyle [mailto:[email-masked]]
> >      Sent: Tuesday, July 09, 2002 4:08 PM
> >      To: Aleksandr Ryabin; Christian Groves; [email-masked]
> >      Cc: Dan Elbert; 'Rajeev K R'
> >      Subject: RE: [Megaco] MGC-MG dialogue failure detection
> >
> >      Doesn't this open the possibility that a secondary MGC could take
over
> >      from a primary that isn't out of service, causing control
conflicts?
> >      If the primary is in service, do we really want a secondary taking
> >      over?
> >      Kevin
> >
> >      -----Original Message-----
> >      From: Aleksandr Ryabin [mailto:[email-masked]]
> >      Sent: Tuesday, July 09, 2002 1:08 PM
> >      To: Christian Groves; [email-masked]
> >      Cc: Dan Elbert; 'Rajeev K R'
> >      Subject: Re: [Megaco] MGC-MG dialogue failure detection
> >
> >      Hi Christian,
> >
> >               This still would not solve "software" fail over (e.g. MGC
with
> >      different IP/port)
> >      as it only will work for "hardware" (hot swap) fail over (e.g. MGC
with
> >      same IP/port),
> >      but then it has nothing to do with the protocol, since as you
correctly
> >      pointed out
> >      MG would not have to know about it.
> >
> >               What I'm trying to find a solution for is a protocol
supported
> >      way
> >      for such
> >      a situation. That is if we can put into MEGACO some provisions to
> >      overcome
> >      this bottleneck in the protocol itself.
> >
> >               My proposals do not deal with "hot swap" mechanism since
"hot
> >      swap"
> >      does not have a problem as you correctly stated. The fact is that
the
> >      protocol
> >      already have a handoff mechanism which already works in most
scenarios.
> >      By
> >      allowing initiating handoff from secondary MGCs would solve the
> >      bottleneck.
> >               The problem with this as I understand is that generally
> >      speaking
> >      MG is
> >      the one that must initiate the control connection (specifically in
case
> >      of
> >      TCP).
> >      This also can be worked around by allowing MG to initiate control
> >      connections
> >      to all MGCs including secondary on startup, or some other time,
before
> >      the primary MGC failure occurs. Only one will be "active", the one
to
> >      current primary MGC thus control connection will be up by the time
> >      secondary MGC will need to send handoff to itself.
> >               Of course this all can be defined as optional
functionality to
> >      be
> >      backward compatible.
> >
> >               Does anybody have any strong objections on allowing
secondary
> >      MGCs to send handoff for itself and/or for MG to establish multiple
> >      control
> >      connections?
> >
> >      Thanks
> >               Aleks
> >
> >      At 09:00 PM 7/8/2002, Christian Groves wrote:
> >      >Hello Aleksandr,
> >      >
> >      >For the problem you describe I assume that you are talking about a
> >      very large
> >      >gateway (that is highly reliable) that would have many new call
> >      attempts
> >      >during
> >      >the 10sec period.
> >      >
> >      >You can address the situation below in a number of ways.
> >      >o Design your MGC so that it hot swaps to another MGC without the
MG
> >      ever
> >      >knowing. ie. use the same connection to the MG. This assumes that
> >      there would
> >      >have to be sharing of states/records between the 2 MGCs. This
would
> >      eliminate
> >      >the problem completely.
> >      >o Use a reliable transport ie. SCTP and set your timers quite
tightly.
> >      This
> >      >would minimise the problem.
> >      >
> >      >Through the development of Megaco/H.248 we stayed clear of using
it as
> >      a "hot
> >      >swap" mechanism as this would lead us to make assumptions about
the
> >      >communication between MGCs which is rather out of scope of our
> >      charter.
> >      >
> >      >Regards, Christian
> >      >
> >      >Aleksandr Ryabin wrote:
> >      > >
> >      > > Hi,
> >      > >
> >      > >          One problem that inactivity package does not solve is
a
> >      service
> >      > > interruption.
> >      > > That is if a primary MGC fails (crashes) such that it can not
send
> >      SC
> >      > > handoff to
> >      > > MG and secondary MGC takes over right away then currently there
is
> >      no
> >      > way for
> >      > > secondary MGC to let MG know that it is an active MGC now,
since SC
> >      has
> >      > to be
> >      > > initiated from MG before any message is sent from new MGC. Thus
> >      using
> >      > > inactivity
> >      > > package even if inactivity timer set to a minimum (10 msec)
there
> >      will be
> >      > > still a service
> >      > > interruption delay of at least Notify time-out period which
counted
> >      in
> >      > > seconds and
> >      > > which is a huge delay for real time systems. That is during
this
> >      time
> >      > > (seconds) an
> >      > > MGC acting as soft switch will not be able to establish new
calls
> >      as well
> >      > > as modify
> >      > > calls currently in progress.
> >      > >
> >      > > One way to work around is to allow any provisioned MGC (e.g.
> >      primary,
> >      > > secondary,
> >      > > etc.) to send SC handoff with its own MgcIdToTry at any point
of
> >      time to
> >      > > take over.
> >      > >
> >      > > Another way could be to extend inactivity timer package and add
> >      Notify
> >      > > time-out ("nto")
> >      > > parameter to "ito" event with the same attributes as "mit"
> >      parameter which
> >      > > will allow MGC
> >      > > to set time-out for Notify response which is send on "mit"
> >      expiration. This
> >      > > way MGC at
> >      > > least will have control of maximum service outage a secondary
MGC
> >      should
> >      > > expect on
> >      > > a primary MGC permanent failure. More over it could be
specified if
> >      "nto"
> >      > > parameter is
> >      > > set to 0 then it would direct MG to start failover process as
> >      described in
> >      > > par 11.5 right
> >      > > away on "mit" expiration without sending Notify to primary MGC.
> >      > >
> >      > > Any comments, or other ideas regarding this problem?
> >      > >
> >      > > Thanks
> >      > >          Aleks
> >      > >
> >      > > At 03:17 AM 7/7/2002, Dan  Elbert wrote:
> >      > > >Hi Rajeev
> >      > > >This are exactly the problems that the package is trying to
solve.
> >      > > >If MGC detects that MG is not responding, just don't send any
> >      message
> >      > to it
> >      > > >until it registers again.
> >      > > >Then when inactivity timer expires, MG send event, doesn't get
a
> >      response,
> >      > > >and finally registers again.
> >      > > >Then same will happen to every MG when the MGC goes down and
up
> >      again.
> >      > > >
> >      > > >Dan Elbert
> >      > > >RADVISION
> >      > > >
> >      > > >-----Original Message-----
> >      > > >From: Rajeev K R [mailto:[email-masked]]
> >      > > >Sent: Saturday, July 06, 2002 4:18 AM
> >      > > >To: [email-masked]; [email-masked]
> >      > > >Subject: Re: [Megaco] MGC-MG dialogue failure detection
> >      > > >
> >      > > >
> >      > > >Hi Jean-Luc,
> >      > > >
> >      > > >ITU-T SG16 has published a document named
> >      > > >H.248 Annex M.6: Inactivity Timer Package
> >      > > >which can be used by MG to detect the MGC failure.
> >      > > >MG can send Notify with event=it/ito and chcek whether MGC
gives a
> >      > response.
> >      > > >When there is no response for some number of attempts, MG can
> >      initiate
> >      > > >registration procedure.
> >      > > >
> >      > > > From the MGC side the package suggest the use of Audit for
ROOT
> >      with
> >      > empty
> >      > > >descriptor.
> >      > > >
> >      > > >But there is another problem, the standard still does not
address.
> >      This is
> >      > > >more from a practicality
> >      > > >point of view. This is from the MGC side. Suppose the MGC
detect
> >      that
> >      > the MG
> >      > > >is not responding and
> >      > > >mark that this MG is unregistered.
> >      > > >Now how can the MGC inform the MG that its registration is
removed
> >      ?
> >      > > >If the MG is actually down there is no problem because when
the MG
> >      > comes up
> >      > > >again, it will register.
> >      > > >If the MG is not down, it will never register again.
> >      > > >When it tries to initiate a call, it will get a response that
the
> >      MG
> >      > is not
> >      > > >registered.
> >      > > >Some MGs register again when they get this response, but some
MGs
> >      do not.
> >      > > >
> >      > > >Another scenario is where all the MGs are registered, but the
MGC
> >      go down
> >      > > >and come up.
> >      > > >How can the MGC force the MGs to register again ?
> >      > > >Is the MGC allowed to use the previous state of
registation(state
> >      > before it
> >      > > >went down) for its MGs ?
> >      > > >
> >      > > >Regards,
> >      > > >Rajeev
> >      > > >
> >      > > >
> >      > > >
> >      > > >
> >      > > >
> >      > > >----- Original Message -----
> >      > > >From: <[email-masked]>
> >      > > >To: <[email-masked]>
> >      > > >Sent: Friday, July 05, 2002 3:57 PM
> >      > > >Subject: [Megaco] MGC-MG dialogue failure detection
> >      > > >
> >      > > >
> >      > > > >
> >      > > > > I would like to point out an issue which is as far as I
know is
> >      not
> >      > solved
> >      > > > > in RFC3015.
> >      > > > >
> >      > > > > I suppose that the  MG is registered to the MGC and that
there
> >      is no
> >      > > > > transaction on going.
> >      > > > > If the MGC fails, there is no way for the MG to detect it
when
> >      > using UDP.
> >      > > > >
> >      > > > > I have already seen two mechanisms implemented by stack
vendors
> >      to
> >      > solve
> >      > > > > this issue :
> >      > > > >
> >      > > > > 1- Use of ping to detect that the correspondent is no more
> >      reachable
> >      > > > > 2- Use of a ServiceChange ROOT with termination = undefined
as
> >      > a  polling
> >      > > > > mechanism
> >      > > > >
> >      > > > >      * on MGC side if no response to the poll, return to
> >      initial state
> >      > > > > waiting for a ServiceChange
> >      > > > >         from the MG. Inform the Call Control that  the
> >      terminations
> >      > are no
> >      > > > > more usable
> >      > > > >         to establish new calls
> >      > > > >
> >      > > > >      * on MG side if no response to the poll, return to
initial
> >      > state and
> >      > > > > try to retrieve the dialog
> >      > > > >         by sending ServiceChange to the primary MGC  or to
the
> >      > secondary
> >      > > > > MGC if it is provisioned.
> >      > > > >
> >      > > > > I think that to avoid that everyone defines it own
mechanism
> >      which can
> >      > > >lead
> >      > > > > to interoperabilty problems
> >      > > > > it should be defined in the RFC.
> >      > > > >
> >      > > > >
> >      > > > > Jean-Luc Lecuyer
> >      > > > >
> >      >
> >
--------------------------------------------------------------------------
> >      > > >------------------
> >      > > > > Jean-Luc Lecuyer :          VND/E10 VS  BU R&D/CCPD Lannion
> >      > > > > Full telephone number :
> >      33
> >      > (0)2 96
> >      > > >04 77 09
> >      > > > > Full fax number :
> >      > > >33 (0)2 96 04 72 35
> >      > > > > Email :
> >      > > >[email-masked]
> >      > > > >
> >      >
> >
--------------------------------------------------------------------------
> >      > > >-------------------
> >      > > > >
> >      > > > >
> >      > > > >
> >      > > > > _______________________________________________
> >      > > > > Megaco mailing list
> >      > > > > [email-masked]
> >      > > > > https://www1.ietf.org/mailman/listinfo/megaco
> >      > > > >
> >      > > >
> >      > > >
> >      > > >
> >      > > >_______________________________________________
> >      > > >Megaco mailing list
> >      > > >[email-masked]
> >      > > >https://www1.ietf.org/mailman/listinfo/megaco
> >      > > >
> >      > > >_______________________________________________
> >      > > >Megaco mailing list
> >      > > >[email-masked]
> >      > > >https://www1.ietf.org/mailman/listinfo/megaco
> >      > >
> >      > > _______________________________________________
> >      > > Megaco mailing list
> >      > > [email-masked]
> >      > > https://www1.ietf.org/mailman/listinfo/megaco
> >
> >      _______________________________________________
> >      Megaco mailing list
> >      [email-masked]
> >      https://www1.ietf.org/mailman/listinfo/megaco
> >

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco

---

From: Aleksandr Ryabin <[email-masked]>
Date: Wed, 10 Jul 2002 13:18:23 -0400

Hi,

         I have to agree with Christian that secondary MGC can always
audit instead of synchronizing with primary MGC and thus secondary
incidentally taking over might not be good for such MGCs since
secondary is not aware of the state of primary MGC. So for such
configurations it does make sense not to allow "secondary handoff".

         But if we are taking about "high density" then MGCs must
have some synchronization mechanism as it will not be an acceptable
delay for such a system to wait for MGC to gather the state information
using audits (it might take a while).
         Also I do not agree that software based "fault tolerance" for MGC
is not practical. A properly designed system with software based
"fault tolerance" can be 4 nines. Thus if secondary MGC is in a
"stand by" mode it can take over immediately when it detects
primary MGC failure, and yet the protocol currently is a bottleneck
for this.

         Perhaps to satisfy both cases MG should have a read/write
"acceptSecondaryHandoff" ROOT termination property (for example,
in inactivity package) which will direct MG to accept, or not to accept
handoff initiated from secondary MGCs. This way MG can be configured
for a specific configuration depending on type of controlling MGCs.
Since this property will exist for MGs that support this package
this approach is backward compatible and will not break any current
implementation.

Comments, please?

Thanks
         Aleks

---

From: Aleksandr Ryabin <[email-masked]>
Date: Wed, 10 Jul 2002 13:18:23 -0400

Hi,

Comments, please?

Thanks
         Aleks