Subject: [Sipping] Overload solutions comments

From: "Terpstra, Rich" <[email-masked]>
Date: Thu, 27 Jul 2006 14:06:54 -0600

I don't remember if the working group came to a consensus regarding
hop-by-hop versus end-to-end load values for the two different
solutions. When reviewing the hop-by-hop approach, I thought of a third
possible approach, which is to return the value of the most loaded
element to the UAC. This solves a couple of problems that the hop-by-hop
doesn't address and keeps the message smaller than the end-to-end
approach. It also solves the problem of intermediary proxies not
supporting the header, as long as they pass it upstream unchanged. The
busiest element approach allows each hop to compare the value in the
header to its own load value. If the proxy's load is higher, it places
itself into the header. Otherwise, it leaves the header as-is and passes
it upstream. The UA can decide what to do with the information as well
as the hop before the congested element.

The problem I have that the hop-by-hop approach doesn't solve is that as
an operator, I see that congestion usually occurs at the UAS and not at
the intermediary proxies. A Network normally has multiple routes to
reach a UAS through diverse proxies, so the proxies themselves rarely
congest. The UAS, however, can become a bottleneck that with the
hop-by-hop approach could cause a domino affect to the intermediary
proxies. The end-to-end approach allowed the UAC to know that the UAS
was congested, so UAC could choose to release some sessions destined for
the UAS, which would avoid congesting the intermediary proxies. I
believe reporting the most congested element will also allow the UAC to
avoid choosing some or all routes based on which element is congested.

Consider the following example to illustrate how it appears that
hop-by-hop doesn't help alleviate congestion, but in fact acerbates it:

+-----+    +-------+    +--------+   +--------+            
| (A) |--->|  (B)  |--->|   (C)  |-->|  (D)   |\              
+-----+    +-------+    +--------+   +--------+ \            
 PSTN          |                                 \+------+      +-----+
               |                                  |  (F) |----->| (G) |
               |                                 /+------+      +-----+
               |        +--------+   +--------+ /
               +------->|  (H)   |-->|  (E)   |/
                        +--------+   +--------+

SIP ------------------------------------------------------------------>

(A) - Phone		(E) - Proxy
(B) - UA		(F) - B2BUA
(C) - Proxy		(G) - Phone
(D) - Proxy       (H) - Proxy

A phone (A) in the PSTN calls a phone (G) on a SIP network. The PSTN
routes the call to a carrier who owns the number for phone (G) via a UA
(B). The carrier determines that there are two diverse routes through
the carrier's network for reaching phone (G). Carrier sends call from UA
(B) to proxy (C).  Proxy (C) sends call to proxy (D). Proxy (D) sends
call to the Feature Server (F) where phone (G) is registered. Call
arrives at B2BUA (F), but B2BUA (F) is overloaded. B2BUA (F) will
complete the call to phone (G), but will report its load to proxy (D).
Proxy (D) will pass its load back to proxy (C). Proxy (C) passes its
load back to UA (B). Let's say that B2BUA (F) Load = 80, proxy (D) Load
= 20, and proxy (C) Load = 30.

A new call comes into the same UA (B) destined for the same B2BUA (F).
UA
(B) sends call to proxy (H). Proxy (H) sends to proxy (E). Proxy (E)
does not know about the load on B2BUA (F), so proxy (E) sends call to
B2BUA (F). B2BUA (F) completes call to phone (G). B2BUA (F) reports its
load now as
85 to proxy (E). Proxy (E) reports its load of 20 back to proxy (H).
Proxy (H) sends back its load of 30 to UA (B).

A third call comes into UA (B) destined for same B2BUA (F). UA (B) sends
call to proxy (C). Proxy (C) sends call to proxy (D). Proxy (D) sees
B2BUA
(F) is loaded at 80, so it chooses not to offer the call to B2BUA (F).
Proxy (D) releases the call with 503 to proxy (C). Proxy (C) sends back
503 to UA (B). UA (B) has a second route to B2BUA (F), so it sends call
to proxy (H). Proxy (H) sends call to proxy (E). Proxy (E) sees B2BUA
(F) is overloaded, so it releases call with 503 to proxy (H). Proxy (H)
sends
503 back to UA (B). UA (B) fails the call back to the PSTN. 

As you can see from this simple example, if B2BUA (F) load had been
relayed back to UA (B), UA (B) could have applied a gapping algorithm to
the call to avoid even trying to reach the B2BUA (F). It becomes even
more important when there are multiple UA (B) that could send to the
B2BUA (F), because then each UA can apply a gapping algorithm
independently in order to alleviate the load on the B2BUA (F). Without
knowing the load of the B2BUA (F), you effectively create a domino
affect such that as more calls come in, the network will offer more
calls to the downstream proxies fronting the B2BUA (F), which will
eventually overload one or both downstream proxies. The proxy (C) will
then start gapping calls to each proxy (D/E), until the proxy (C)
becomes overloaded. At this point, the UA (B) can start call gapping to
alleviate the proxy (C). Once proxies
(C/H) are no longer overloaded, UA (B) will start sending to proxy (C)
again. The proxies (D/E) may still be overloaded. If so, the proxy (C)
will eventually go back into an overloaded state.

As you can see, the hop by hop didn't really help with overload
condition. In fact, it compounded the problem by overloading
intermediate elements that would not have become overloaded if each
element had relayed the B2BUA (F) load to UA (B).

A real world example of this behavior is for a Customer that offers
Pre-Paid Calling Card. Then a holiday like Mother's Day comes around and
everyone tries to use the Calling Cards that day. The B2BUA (F) can only
sustain 1000 calls simultaneously, but the company has sold a million
cards. For the 3 minutes a call is up, B2BUA (F), proxies (D/E), proxies
(C/H), or UA (B) gap the calls. However, calls continue to arrive at
many UA's (B) across the country as the 900,000 callers that haven't
gotten through continue to hit redial on their PSTN phones. This one
event has overloaded the Customer B2BUA (F) and proxies (D/E). This
application in addition to other traffic have overloaded the proxy (C).
It may even be overloading many UA's (B). However, if all UA's (B) had
known how busy the B2BUA (F) was, each UA (B) could have
alleviated/avoided overloading the UA (B), proxies (C/H), and proxies
(D/E). UA (B) could have also freed up the PSTN circuits quicker, which
may have resulted in calls to other destinations that would have
prevented the PSTN from offering as many calls to the overloaded B2BUA
(F).

Rich Terpstra
 
Level 3 Communications
Softswitch Architecture
1025 Eldorado Blvd, Broomfield, CO, 80021
Office: (720) 888-2994

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Volker Hilt <[email-masked]>
Date: Fri, 28 Jul 2006 00:31:34 -0400

Hi Rich,

thanks much for the comments! More inline.

> as the hop before the congested element.
The main problem with the busiest element approach is that it can cause 
a proxy to falsely report overload even though it is not overloaded at 
all and could still forward messages to other non-overloaded proxies. 
Here is an example:
                      95
                     +---+
           10   95/--| C |
+---+ 95 +---+ /    +---+
| A |----| B |       10
+---+    +---+ \    +---+
                10\--| D |
                     +---+

In this example, C is overloaded and reports a load 95 to B. D is fine 
and reports a load of 10. In the busiest element approach, B would 
report a load of 95 to A (since 95 is higher than its own value of 10). 
Thus, B would effectively report overload and ask A to stop forwarding 
traffic to it even though it is doing fine and proxy D can still be 
reached through B. Essentially, one overloaded leaf node in a 
distribution tree would shut down the traffic to all other nodes in the 
tree by reporting overload that is passed upstream.

Another approach is one that works entirely on an end-to-end basis 
(i.e., only endpoints report load). However, this also has its problems. 
Assume that in the above example B is running at a load of 95 whereas C 
and D are running at 10. If only the load of C and D is reported 
upstream, A would not become aware of the overload in B and would not 
throttle traffic.

The only way to make this work seems to be a path-based overload 
control, in which each proxy along the path to a certain target reports 
its load upstream. The problem with this is that it is complex and very 
hard to do right. This approach would require large load reports (one 
load sample for each proxy on the way) and it seems very hard for a 
receiver of such a multi-sample report to determine the correct action 
based to this feedback information. E.g., by how much should it throttle 
traffic, which destinations are affected, how are multiple paths to the 
same destination handled, ...

> the intermediary proxies.

A UAS can use overload control the same way as a proxy. I don't see a 
UAS specific problem here.

> A Network normally has multiple routes to
> proxies. 

The upstream neighbor of an overloaded UAS does not get clogged by the 
messages it can't deliver. When it receives an overload indication and 
does not have an alternative route for the target it will start to 
reject messages instead of forwarding them (e.g. using a reasonable 
retry after value). Only when the traffic increases further to a point 
where even the proxy can't reject all messages any more, it will send an 
overload indication to its upstream neighbor and ask the neighbor to 
start throttling load.

> The end-to-end approach allowed the UAC to know that the UAS
This is actually the correct behavior since F has reported overload to 
its upstream neighbors.

I think what you are trying to do is to optimize the protocol in a 
certain scenario. My concern is that such an optimized solution will end 
up being incredibly complex and very hard to get right and to manage.

Thanks,

Volker

-- 
Volker Hilt                      Bell Labs / Lucent Technologies
phone: +1 732 332 6432           101 Crawfords Corner Rd
email: [email-masked]     Holmdel, NJ 07733

---

From: "Hadriel Kaplan" <[email-masked]>
Date: Fri, 28 Jul 2006 00:48:48 -0400

Hi Rich,
Doesn't that imply that UA (B) knows all future pre-paid calling card calls
will go to B2BUA(F), and only (F)?  

My point is proxies can find alternate routes, fork, etc.  If you give (F)'s
congestion back to (B), and he does something with it, you bypass the
intermediate proxies' capability to send the call to B2BUA (F') (a secondary
feature server you don't have on your drawing).  UA(B) will not know if
there is or is not a (F'), because it depends on the owner of (F) or
Proxy(D).  So unless UA(B) knows for all possible AoR's what all possible
final UAS are, it will be gapping when it shouldn't.  

It's just too topology/case specific methinks.

For example what if this was a call center number instead, with hunting.  Or
what if the calls were going the other way (to the PSTN) and the PSTN
Gateway was getting congested.

Although I guess if the intermediate Proxies only return the congestion
notification for cases where they have no alternate path for that
AoR/request-uri, it may work, but it means the UA(B) keeps a list of such
congested AoRs.  I think that may get complicated, fast.

I know the pre-paid card number example is real, but to me it's more an
argument/case for code gapping than system overload/congestion control.  
My 2 cents.

-hadriel

> -----Original Message-----
> From: Terpstra, Rich [mailto:[email-masked]]
> Sent: Thursday, July 27, 2006 4:07 PM
> To: [email-masked]; [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: [Sipping] Overload solutions comments
>

---

From: Daryl Malas <[email-masked]>
Date: Fri, 28 Jul 2006 11:19:49 -0600

I think Rich's point introduces another problem to solve.  If I
understand it correctly, if the UA or Registrar (whether one or many)
associated with the end point is overloaded, I can avoid sending the
request clogging up interim elements waiting for hop-by-hop overload
backup.

I almost think you would have to enable some type of a "INFO" request to
the destination UA or registrar to determine if enough resources exist
to send the request thereafter.  It's somewhat of a RSVP method in SIP.
Otherwise, the requesting UA would have to know the topology to reach
the destination UA in order to understand whether the highest congested
element is one that would ultimately impact sending the request via a
specific route.  I think this latter option will be quite complex.

In either case, I think it's an interesting problem to try and resolve
with whatever the final solution is.

Thanks...--Daryl