Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to solve a network equipment failure scenario

From: "Schantz, John" <[email-masked]>
Date: Fri, 28 Jun 2002 13:32:43 -0500

Javier,

I think you are giving multi-homing more credit than it is due.  The
scenario happens whether multi-homing is implemented or not.  And it is
not just my problem, it's a general problem.  The issue is that the AS
or SG that is not next to the failure can not detect it via M3UA or SCTP
until several seconds (and lots of traffic) has passed.  Why not let the
side that can immediately detect the problem inform the side that can't
so the other associations or multi-home IP addresses can be used to go
around the problem.  Test it in your lab, connect 2 systems via a LAN
and pull the Ethernet Cable from one of the interfaces on one system.
How long did it take the other system to detect the problem and move the
traffic?

John

-----Original Message-----
From: Javier Pastor-Balbas (ECE)
[mailto:[email-masked]] 
Sent: Friday, June 28, 2002 11:30 AM
To: Schantz, John; Javier Pastor-Balbas (ECE); Tolga Asveren;
[email-masked]
Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to solve a network
equipment failure scenario

Hi John, 

See below...

> -----Original Message-----
> From: Schantz, John [mailto:[email-masked]]
> Sent: Friday, June 28, 2002 5:56 PM
> To: Javier Pastor-Balbas (ECE); Tolga Asveren; [email-masked]
> Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to
> solve a network
> equipment failure scenario
> 
> 
> Hi Javier.
> 
> You should not assume that multi-homing is always there.  Not all 
> architectures support it and especially if the NIC only has a single 
> Ethernet interface and the NICs are handled independently and are only

> linked at the M3UA level.  Also NTFY is a very weak message and you 
> can probably get by by ignoring them.  Also aren't they only sent
> by SGs and
> how would indicate the real problem?
> 
Yes.

> We have done a lot of testing and found that if you pull an Ethernet 
> cable (equal to a cable cut, hub or router failure, etc.) the 
> connecting ASP/SGP detects the failure immediately and moves its 
> outbound traffic to other associations.  The other side of the 
> association is left hanging and does not detect the failure very 
> quickly.
> 
> I felt that it would be a simple matter to inform the remote
> side of the
> failure since that would be quicker than waiting for it to detect the
> failure.  If you guys are not interested in solving issues 
> like this, I
> will not waste my time on them.
> 

John, it is not a matter of not being interested in it. In fact, I am.
It is a matter of not having enough information.  :-/

For the normal case (IMHO) anyone will use a multihoming association
with hardware redundancy to get the desired behaviour. But I'm lacking
the knowledge of what are you using in your backbone and how do you use
SCTP and M3UA. At least I didn't undertand it enough....

Basic questions for me are:
- How is your networks set up?
- Do you have two associations between two nodes (SGP/ASP)?

(just getting more info to find out a solution to your problems)

Cheers // Javier.

> John
> 
> -----Original Message-----
> From: Javier Pastor-Balbas (ECE)
> [mailto:[email-masked]] 
> Sent: Friday, June 28, 2002 5:57 AM
> To: Schantz, John; Tolga Asveren; [email-masked]
> Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to 
> solve a network
> equipment failure scenario
> 
> 
> Hi John, 
> 
> You are talking about a hub/router failure:
> 
> Router failure: ideally if you have a router failure you will have
> alternatives paths if the network configuration it's well set up. This
> means that the routing protocols will update the routes and 
> just a small
> delay will happen. But everything keeps working. Multihoming is not
> crucial here.
> 
> Hub failure. For the recommended configuration with multihoming, this
> should not be a major problem. Multihoming allows you to have 
> multi-hub
> configuration. And then, if one hub fails you will still have 
> someothers
> to use with different local NICs.
> 
> What it seems to be clear to me is that multihoming has its 
> advantages.
> You are removing one of the main reliability features included for the
> transport of SS7oIP if you're not using it.
> 
> 
> 
> Regarding your text, I don't understand very well your proposal. ASPID
> is just used by the NTFY message and it is sent with ASPUP to 
> inform the
> SG of its value.
> 
> More comments below....
> 
> 
> 
> > -----Original Message-----
> > From: Schantz, John [mailto:[email-masked]]
> > Sent: Friday, June 28, 2002 12:32 AM
> > To: Tolga Asveren; [email-masked]
> > Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to
> > solve a network
> > equipment failure scenario
> > 
> > 
> > Tolga, we don't support it and the effort to support it 
> would not be 
> > worth it.  I'm from the SS7 side of the world so I believe in true 
> > multi-homing via SS7 like methods.
> > 
> > -----Original Message-----
> > From: Tolga Asveren [mailto:[email-masked]]
> > Sent: Thursday, June 27, 2002 4:13 PM
> > To: [email-masked]
> > Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to 
> > solve a network
> > equipment failure scenario
> > 
> > 
> > John,
> > 
> > > -----Original Message-----
> > > From: Schantz, John [mailto:[email-masked]]
> > > Sent: Thursday, June 27, 2002 4:42 PM
> > > To: Tolga Asveren; [email-masked]
> > > Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to solve a
> > > network equipment failure scenario
> > >
> > >
> > > Tolga, if the HUB or router that the NIC/Ethernet is connected to
> > > fails, it will take ALL association with it.  SCTP 
> > multi-homing will
> > > not help you!
> > [TOLGA] But then how can the other association, from which 
> you propose
> 
> > to send ASPDOWN[ACK] survives? Probably it uses another 
> NIC/Ethernet, 
> > but then why not make this interface also part of the multihomed 
> > association?
> > >
> > > -----Original Message-----
> > > From: Tolga Asveren [mailto:[email-masked]]
> > > Sent: Thursday, June 27, 2002 3:48 PM
> > > To: [email-masked]
> > > Subject: RE: [Sigtran] M3UA: IG: ASP IDs & a proposal to solve a
> > > network equipment failure scenario
> > >
> > >
> > > John,
> > >
> > > > We are pretty mush finished with our development of M3UA
> > and we are
> > > > in
> > >
> > > > the process of trying to find ways to improve or fix some of the
> > > > holes.  I have a proposal below for fixing a network equipment 
> > > > failure
> > >
> > > > scenario that I would like to get added to the IG.  It will also
> > > > clarify the use of ASP IDs. I also think the proposal could be 
> > > > extended to perform a "change-over" if we
> > > > are willing to get some transmission info from SCTP, 
> but that can
> > > wait.
> > > > The failure of IP network equipment (HUBs, routers, etc.) has a
> > > > major impact on the performance of M3UA (and other 
> UAs).  If the 
> > > > failure occurs at
> > > an
> > > > adjacent network interface, the connecting ASP or SGP can detect
> > > > this failure immediately (association failure) and can move the 
> > > > traffic to another association.  The problem is that the 
> > > > non-adjacent ASP or SGP
> > > can
> > > > not detect the failure immediately since it must wait for missed
> > > > SCTP SACKs or Heartbeat Acks.  Section 4.3.4.6 states that if a 
> > > > Heartbeat Ack (or
> > > any
> > > > other message) is not received within 2*T(beat), then the remote
> > > > M3UA
> > > is
> > > > considered unavailable.  There are a couple of problems
> > with this:
> > > > 1)
> > > the
> > > > non-adjacent side is still sending messages during this 
> 2*T(beat)
> > > > time period, this period could be several seconds during which 
> > > > message
> > > buffers
> > > > could overflow and/or messages could be lost, and 2) only the
> > > > ASP/SGP connected to the failure is unavailable while the other 
> > > > ASP/SGPs at the node could still be available and 
> should have the 
> > > > traffic moved over to
> > > them as
> > > > soon as possible..
> > > > Below is a proposal that should solve most of this
> > problem and limit
> > > > (eliminate?) the possible message loss.    The M3UA RFC 
> allows for
> > the
> > > > assignment of IDs to ASPs but they are only transported
> > in the ASP
> > > > UP message and there is no discussion of what they are used for.
> > > > Some changes and clarifications are needed.  First, for 
> > the ASP IDs
> > > > to be useful,
> > > they
> > > > must be unique per association from an AS.  Second, the
> > ASP Down and
> > > ASP
> > > > Down Ack must also have the ASP ID parameter as an optional
> > > > parameter (adding it to the ASP Inactive [Ack] should also be 
> > > > considered). Third, an ASP/SGP that still has an available 
> > > > association to the remote node should be
> > > > able to send (forward) an ASP Down/ASP Down Ack that 
> contains the
> > > > ASP ID of
> > > > the unavailable ASP (association).  Each of these items are
> > > > addressed below.
> > > > ASP ID uniqueness
> > > > It is fairly easy to show that assigning an ASP ID to 
> the process
> > that
> > > > handles all associations for a network interface is of 
> very little
> > > use.
> > > > Suppose that you have an ASP that has an association to 2 (or
> > > > more) SGPs at
> > > > an SG.  Suppose the ASP only wants to stop traffic on one of the
> > > > associations.  It would send an ASP Inactive or ASP Down. 
> >  There is
> > > > no
> > > ASP
> > > > IDs present but there is an implied ASP ID for the ASP.
> > Will the SG
> > 
> > > > assume that all associations to this ASP are going out of
> > service or
> > > > should it only
> > > > assume that the association that carried the message is
> > going out of
> > > > service?  It is the latter case that is assumed.
> 
> I would say here that is the ASP who sent the message the one 
> who moves
> to DOWN (out of service). But assocation could be still used by other
> ASes
> 
> > > > Hence, the
> > > > generic ASP Up,
> > > > ASP Inactive, ASP Down, etc. only apply to the association (ASP 
> > > > and ASP ID) that the messages are received on.  Therefore, the 
> > > > implicit and explicit ASP
> > > > IDs must be unique per association between the AS and SG.
> 
> I would say ASPID is unique per ASP and AS
> 
> > > > ASP ID Optional Parameter in ASP Down and ASP Down Ack The need 
> > > > for the ASP ID to be an optional parameter in the ASP 
> Down and ASP
> > > > Down Ack will become clear in the next section.  It 
> will allow the
> > > ASP/SGP
> > > > to be explicit about which association is being impacted.
> 
> I don't understand the real need for this...
> 
> > > > Making
> > > > the ASP ID optional in the ASP Inactive and ASP Inactive 
> > Ack should
> > > > also be considered.
> > > > Forwarding of Locally Detected Association Failure.
> > > > When a node detects an association failure, its M3UA 
> should notify
> > the
> > > > affected adjacent node of the failure via an available 
> association
> > to
> > > the
> > > > adjacent node.
> 
> If you just have one association and it is broken....
> 
> It seems to me that you have two association between an SGP 
> and an ASP.
> Is that true? And you'd like to inform using the second assoc that the
> first assoc failed, right?
> 
> I'm afraid this is not the standard use for ASPID. ASPID 
> should identify
> an ASP.
> 
> Another possibility is that you consider your ASP as two ASPs 
> and set an
> ASPID for each one. In that case you could sent NOTIFY(ASPID) with the
> ASPID that is failing through the working association.
> 
> Cheers // Javier.
> 
> 
> > > >When the AS/SG detects the association failure, it
> > > should
> > > > send an ASP Down [Ack] with the "failed" ASP ID
> > (association) on an
> > > > association that is still available to the adjacent SG/AS.  When
> > > [TOLGA] If there is still a healthy association between those two
> > > nodes, why initially not to have a multihomed SCTP 
> > association, which
> > > also includes this interface? If number of interfaces is high, IMO
> > > this would increase detection time for host failures, but 
> > is this the
> > > reason, why you didn't prefer to use multihoming?
> > > > the M3UA at
> > > > the adjacent SG/AS receives the ASP Down [Ack] with an
> > ASP ID that
> > > > is different from the known ASP ID for the association, 
> it should
> > > > handle message as though the ASP Down [Ack] was received by the 
> > > > SGP/ASP
> > > > (association) corresponding with the ASP ID.  This will 
> cause the
> > > > SG/AS transition the ASP/association to the DOWN state 
> and to move
> > the
> > >
> > > > traffic away from the failed ASP/association (that it 
> has not yet
> > > > detected).
> > >
> > >
> > > _______________________________________________
> > > Sigtran mailing list
> > > [email-masked] https://www1.ietf.org/mailman/listinfo/sigtran
> > 
> > 
> > _______________________________________________
> > Sigtran mailing list
> > [email-masked] https://www1.ietf.org/mailman/listinfo/sigtran
> > 
> > _______________________________________________
> > Sigtran mailing list
> > [email-masked] https://www1.ietf.org/mailman/listinfo/sigtran
> > 
> 

_______________________________________________
Sigtran mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/sigtran

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 29 Jun 2002 00:01:54 +0200

John,

I think you are using the SCTP parameters as given in RFC 2960. This is
not appropriate for SS7/IP transport which does not take place over the
Internet. You have to use different setting.

Andreas Jungmaier and myself have done some experiments showing that you
can perform a failover in the limits given by the SS7 specs. Due to
the copyright stuff we can not put it on the web until we have presented
it. it. We will put it on www.sctp.de at the end of July.

The parameters to modify are Path.Max.Retransmit and RTO.min and RTO.max.

Best regards
Michael

>>>>> traffic away from the failed ASP/association (that it
[email-masked]

---

From: "Tolga Asveren" <[email-masked]>
Date: Fri, 28 Jun 2002 18:17:03 -0400

John,

> Sent: Friday, June 28, 2002 2:33 PM
[TOLGA] IMO, for M3UA SCTP multi-homing is a good choice to deal with
network problems. Even if it might take several seconds to detect a
path-failure -based on the values of SCTP parameters- still no message will
be lost when traffic is switched to the other path. The real problem case
is, when there is a host failure. In such a case, multihoming won't help,
nor will changeover-like mechanisms, so this is also a problem case for
traditional SS7. But most probably with SCTP there will be more message loss
in such a situation, based on the actual value of T3. I think to handle this
situation, corid draft of Brian provides a good strategy (maybe something
similar could be done on SCTP level to provide a more generic solution(?))
BTW, for the scenario you described -assuming that both SCTPs are configured
with the same parameter values- I would say both sides should detect
association failure more or less at the same time.

---

From: Al Varney <[email-masked]>
Date: Fri, 28 Jun 2002 18:30:59 -0500

At 06:17 PM 6/28/02 -0400, Tolga Asveren wrote:
>be lost when traffic is switched to the other path.

    Unfortunately, SS7 isn't TCP -- message loss (at a low rate) is 
acceptable in SS7.  What is less acceptable is excessive delay.  [TCP 
accepts delay and tries to prevent message loss.]  Unfortunately,  IP can 
do nothing to address delay.  Only the engineering of the network and the 
intrinsic network reliability and failure handling can do it (below the IP 
layer), with virtually no assistance from the IP protocol layer.  Perhaps 
MPLS or other end-to-end "connection-oriented transport" can fix the delay 
problem, but SCTP must (at least in RFCs) assume a generic "Internet".

    So ... SCTP + M3UA assures no message is lost over the several-second 
path hunt for a working association.  Unfortunately, in doing so it doesn't 
assure that no CALLS are lost as a result.  An IAM (requesting a continuity 
test) delayed by 2 seconds will almost always result in an IAM 
re-attempt.  Another 2-second delay could fail the call entirely.  Plus it 
ties up connections waiting on timers to expire.  SCCP/TCAP delays to SCPs 
are often limited to about 3 second round-trip delays -- and a call failure 
after that time-out.

    Worse, telephone users themselves have been trained (by decades of 
call-setup-time reductions) to just hang up if audible ring tone is delayed 
by more than a few seconds.  How long do you wait on a call?  So SCTP 
"re-homing" due to spotty network congestion could not only cause some call 
attempts to be dropped by call servers, but could also trigger lots of 
abandons by callers (I've found adjusting THEIR timer values rather 
difficult).  And the side-effect of lots of user abandons?  They try the 
call AGAIN!  And thus increase the short-term signaling traffic load.  The 
last thing you need during those events are "old messages" eating up your 
messaging resources (buffers, SCTP associations, transmission facilities, 
etc.).

    SCTP/M3UA would be much better off if "old messages" (say, queued for 
more than a second or two) were just discarded from transmit buffers.  It 
takes a rare combination of multiple failures for the traditional SS7 
network to delay messages more than a couple of seconds.  The higher-layer 
protocols (ISUP, IN, etc.) are built on the assumption that messages may be 
occasionally lost, but delayed very rarely.  Late-arriving messages can 
interfere with other calls -- lost ones usually affect at most one call.

>  The real problem case
>traditional SS7.

    Traditional SS7 hosts rarely fail, and provide multiple independent 
interfaces to independent network elements/interfaces.  M3UA hosts probably 
need to do the same.  Unfortunately, SCTP multihoming can't convert 
unreliable hosts and a few NICs into a real SS7 SEP.  Maybe M2PA with 
several NICs, each connected to totally independent private networks, would 
come close to fixing the NIC/network path issue.  The unreliable host issue 
will persist until SS7/ISUP/SCCP are replaced with protocols designed for 
stateless, multiple-copy servers -- assuming such protocols can really 
support the voice call expectations of most users.  But then you wouldn't 
need SCTP....

Al Varney -- just my usual ranting

---

From: Tolga Asveren <[email-masked]>
Date: Sat, 29 Jun 2002 11:13:56 -0700 (PDT)

Al,

IMO, for the delay problem you mentioned neither SCTP
nor any protocol at the same layer can do anything.
One has to have a reasonable IP network for this. But
I don't think this is unachievable, even for Internet
minimum value for min.Rto is given as 1 sec. 

For host failures, the situation might be  worse than
in regular SS7 case, but I think by not letting cwnd
grow too much -probably actual value of T3 and message
throughput should be considered to decide, which value
is "too much" for cwnd at a given time-, one can
decrease number of lost messages to an acceptable
value.

  Tolga

> Al Varney -- just my usual ranting

__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 29 Jun 2002 20:47:58 +0200

Dear all,
see my comments below.
Best regards
Michael

On Saturday, June 29, 2002, at 01:30 AM, Al Varney wrote:

> accepts delay and tries to prevent message loss.]  Unfortunately,  IP 
> can do nothing to address delay.  Only the engineering of the network 
> and the intrinsic network reliability and failure handling can do it 
> (below the IP layer), with virtually no assistance from the IP protocol 
> layer.  Perhaps MPLS or other end-to-end "connection-oriented 
> transport" can fix the delay problem, but SCTP must (at least in RFCs) 
> assume a generic "Internet".
>
>    So ... SCTP + M3UA assures no message is lost over the 
> several-second path hunt for a working association.  Unfortunately, in 
> doing so it doesn't assure that no CALLS are lost as a result.  An IAM 
> (requesting a continuity test) delayed by 2 seconds will almost always 
> result in an IAM re-attempt.  Another 2-second delay could fail the 
> call entirely.  Plus it ties up connections waiting on timers to 
> expire.  SCCP/TCAP delays to SCPs are often limited to about 3 second 
> round-trip delays -- and a call failure after that time-out.
>
> call-setup-time reductions) to just hang up if audible ring tone is 
> delayed by more than a few seconds.  How long do you wait on a call?  
> So SCTP "re-homing" due to spotty network congestion could not only 
> cause some call attempts to be dropped by call servers, but could also 
> trigger lots of abandons by callers (I've found adjusting THEIR timer 
> values rather difficult).  And the side-effect of lots of user 
> abandons?  They try the call AGAIN!  And thus increase the short-term 
> signaling traffic load.  The last thing you need during those events 
> are "old messages" eating up your messaging resources (buffers, SCTP 
> associations, transmission facilities, etc.).
>
>    SCTP/M3UA would be much better off if "old messages" (say, queued 
> for more than a second or two) were just discarded from transmit 
> buffers.  It takes a rare combination of multiple failures for the 
> traditional SS7 network to delay messages more than a couple of 
> seconds.  The higher-layer protocols (ISUP, IN, etc.) are built on the 
> assumption that messages may be occasionally lost, but delayed very 
> rarely.  Late-arriving messages can interfere with other calls -- lost 
> ones usually affect at most one call.
As I have written several times, you have to use others SCTP parameters 
than given in RFC 2960. But
you are right saying that it makes no sense (at least, may it hurts) 
that the SIGTRAN layers still
retransmits messages which are than retransmitted from the SIGTRAN user 
(ISUP) in your case. That is
one of the reasons the ID
http://www.ietf.org/internet-drafts/draft-stewart-prsctp-00.txt
does exist. It allows you exactly what you want to do: You can specify a 
lifetime for your data.
> interfaces to independent network elements/interfaces.  M3UA hosts 
> probably need to do the same.  Unfortunately, SCTP multihoming can't 
> convert unreliable hosts and a few NICs into a real SS7 SEP.  Maybe 
> M2PA with several NICs, each connected to totally independent private 
> networks, would come close to fixing the NIC/network path issue.  The 
> unreliable host issue will persist until SS7/ISUP/SCCP are replaced 
> with protocols designed for stateless, multiple-copy servers -- 
> assuming such protocols can really support the voice call expectations 
> of most users.  But then you wouldn't need SCTP....
>
SCTP helps you in case of network failures, not in case of host 
failures. For handling host failures
the adaptation layers use the ASP/SGP stuff. There is even a whole WG 
(Reliable Server Pooling) dealing with
this problem. And guess, the people working in the RSerPool WG are 
mainly from the SIGTRAN WG...