Subject: RE: [Speermint] Congestion and Load Balancing

From: "Michael Hammer \(mhammer\)" <[email-masked]>
Date: Wed, 16 Aug 2006 12:03:15 -0400

Inline. 

> -----Original Message-----
> From: Brian Rosen [mailto:[email-masked]] 
> Sent: Wednesday, August 16, 2006 11:12 AM
> To: Michael Hammer (mhammer); [email-masked]
> Subject: RE: [Speermint] Congestion and Load Balancing
> 
> I think peering is functionally a star: peers have some 
> collection of elements facing the peering fabric and peering 
> goes from one edge to the other.  The "fabric" may be nothing 
> but an IP network.  It may even be the Internet.  There may 
> or may not be any active element at the hub of the star.  If 
> there is, it's a proxy server.  If there isn't, it's just an 
> IP switch or functional equivalent.

OK

> If the federation, by contract, limits the total number of 
> calls towards the fabric from any one peer, and either 
> federation contract policy or bilateral contract policy 
> limits traffic between two specific peers, I think we can 
> create non-blocking peering (or at least non blocking to the 
> point where the incidence of blocking is low enough that we 
> can ignore it and busy the call).  I think that is sufficient.

So, every time there is a new peer we need to renegotiate all contracts?

How do we can figure out how much traffic can be exchanged between two
peers statically?  
Then, next question is what factors would cause us to change that?  
Then, is there any reason this can not be adjusted dynamically?

I am not worried about the IP fabric between the peers as that is an IP
network congestion problem.  My issue is "congestion" at the receiving
point.  Are we still addressing the same problem?

> The SIP overload work is dealing with the 503 issue.  I don't 
> have anything to offer beyond that work, do you?
> 
> I am extremely suspicious of simplistic congestion control 
> mechanisms that try to push congestion knowledge back earlier 
> in a network.  

Are you talking L3 or L5 network congestion here?

I was thinking that we are talking about knowledge being passed from one
peer to one adjacent peer, that is, only one hop away.

> We have LOTS of experience that getting them 
> right is not easy.  Look at the archives on "Explicit 
> Congestion Notification" in TCP for example.  

Can you explain the analogy here?  Does ECN address pair-wise traffic
levels, or is it more concerned with cross-traffic in the fabric between
the peers.  There are a lot of unkowables in TCP that don't exist in
this case.  Also, the scale at which TCP needs to operate is much
larger.

> Unless you can 
> show me a big network successfully using SUB/NOT to push 
> congestion state backwards, then I don't believe it will 
> work.  

This is an argument that SUB/NOT should not be used for anything the
first time.

> I guarantee it won't work if it's literally just a 
> notification to stop: that is, there is no explicit text on 
> how you control the rate of messages, and how you control the 
> on/off nature of the mechanism so that it has dampening.  
> Getting such text right, and behavior right at both ends, is hard.
> 
> Brian

I am not saying that it might not be hard, I just don't buy that as a
reason for not trying.  Imagine if you had said that at the advent of
TCP (your words with a few substitutions):

"If the *network*, by contract, limits the total number of 
*packets* towards the fabric from any one peer, and either 
*network* contract policy or bilateral contract policy 
limits traffic between two specific peers, I think we can 
create non-*congestion* peering (or at least non *congestion* to the 
point where the incidence of *congestion* is low enough that we 
can ignore it and *reject the traffic*).  I think that is sufficient."

TCP would not have throttling mechanisms and offered traffic would be
statically limited.

Mike

> 
> > -----Original Message-----
> > From: Michael Hammer (mhammer) [mailto:[email-masked]]
> > Sent: Wednesday, August 16, 2006 10:45 AM
> > To: Brian Rosen; [email-masked]
> > Subject: RE: [Speermint] Congestion and Load Balancing
> > 
> > Brian,
> > 
> > Do we have the same topological picture here?
> > 
> > Your words suggest peering is 1-1 with an SBC on each side of a 
> > boundary, rather than an entry signaling node facing an IP network.
> > 
> > In essence the policy point is acting as a load balancer.  The 503 
> > problem occurs precisely because you may be facing an 
> unknown N number 
> > of federation peers and it is already too late to react 
> when you are 
> > being overloaded with INVITEs.  An INVITE load balancer may be 
> > overwhelmed.  The idea behind the SUB/Notify mechanism is that it 
> > would be orders of magnitude below these spigots of INVITE sources.
> > 
> > Getting real starts with asking the right questions and solving the 
> > same problems.  Thus, my first question above.  We may be 
> envisioning 
> > different approaches to peering.
> > 
> > Mike
> > 
> > 
> > > -----Original Message-----
> > > From: Brian Rosen [mailto:[email-masked]]
> > > Sent: Wednesday, August 16, 2006 9:55 AM
> > > To: 'Khan, Sohel Q [CTO]'; [email-masked]
> > > Subject: RE: [Speermint] Congestion and Load Balancing
> > >
> > > I can't imagine any delay through a peering system that 
> could add up 
> > > to something an application can detect.  It should be 
> very small.  
> > > If it's not, get a new peering system, something is wrong.  Most 
> > > peering is a SIP exchange directly between the peers 
> (with an ENUM 
> > > dip and possibly some other simple database things on the 
> > > origination side).  The transit time TO the peering point should 
> > > swamp the actual signaling exchange.
> > >
> > > What kind of "hot spot" could you imagine we could create in a 
> > > peering system that was not due to poor engineering?  
> Best I can do 
> > > is to force an underpowered SBC to face the peering network and 
> > > bottleneck on that.  Don't do that.
> > >
> > > People, I think we need to get real here.  Let's start throwing 
> > > around realistic numbers, and see if they work or don't.  
> What I see 
> > > here is telephone engineering numbers in an IP environment.
> > >
> > > I believe in millions of BHCA between peers, and gigabits of 
> > > bandwidth.  Is someone believing we are building peering systems 
> > > with hundreds of millions of BHCA between a single pair 
> of peers?  
> > > Hundreds of gigabits of bandwidth don't seem to be a 
> problem for a 
> > > peering system; we can do that with a single element these days.
> > >
> > > I can imagine we WILL get numbers in the hundreds of millions of 
> > > BHCA some day.  I don't think we need to worry about that now.
> > >
> > > I can easily see how you can get beyond a single 
> signaling element 
> > > facing or within the peering fabric, but I think load 
> balancing that 
> > > is pretty straightforward, and doesn't require much 
> standardization.  
> > > I don't think you ever create hotspots in the peering.
> > >
> > > I don't believe in blocking calls before the termination 
> resource.  
> > > That may get congested, but then "Busy" is the 
> appropriate response.
> > >
> > > Brian
> > >
> > >
> 

_______________________________________________
Speermint mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speermint

---

From: "Brian Rosen" <[email-masked]>
Date: Wed, 16 Aug 2006 14:33:39 -0400

In line

> Sent: Wednesday, August 16, 2006 12:03 PM
No.  Some new bilaterals MAY be needed.  I can't imagine any peer LOWERING
any constraints on existing peering arrangements to handle a new peer.  This
is just like IP peering in this regard.

> How do we can figure out how much traffic can be exchanged between two
Are you asking how do peers determine what the actual value should be in a
bi-lateral agreement?  I think that is mostly determined by what the
expectation is between the peers of what "normal" traffic would be, with
some allowance for abnormal.  I think it's accumulated experience, and I
think it's very unsophisticated.  It's more of a limit on the extent
abnormal traffic can be dealt with (for example, are peering with the
network that supports American Idol?)

> Then, next question is what factors would cause us to change that?
Experience

> Then, is there any reason this can not be adjusted dynamically?
Yes.  We don't know how to do that reliably yet, and we don't really need to
make it more complex -- simple mechanisms will work.

> point.  Are we still addressing the same problem?
Yes, although I think "receiving point" has two meanings:
1) The ingress point for the termination peer in the peering system.  I
think this can be made to be large enough that we don't need anything at the
origination side that responds in anything but trivial ways.

2) The actual termination point (or the proxy that directly serves it) in
the termination network.  This really is a busy condition, and can be
reflected back to the originating UA as busy.

> Are you talking L3 or L5 network congestion here?
L5, L3 CAN be dealt with using well understood mechanisms.

> peer to one adjacent peer, that is, only one hop away.
Yes, that is what I understood you to be worried about.  I don't see that as
a problem worth solving = we can build those parts of the network beefy
enough that the probability of congestion is low enough to pretty much
ignore.  Again, I'll admit to the cascading 503, I think the SIP overload
work addresses that, and we can use what they come up with for that problem.

> this case.  Also, the scale at which TCP needs to operate is much
ECN addresses the effect of aggregate congestion on a specific pairwise TCP
connection.  I think that TCP is more complex, because the number of
messages per transaction in SIP is so much fewer than the number of messages
in a typical TCP session.   You typically don't limit how many TCP sessions
can be created in the middle of the network (you do at the endpoints).
However, the point is that reflecting congestion state backwards in the
network, and effectively using that information to control traffic presented
to a specific point in the network, regardless of the exact circumstances,
has proven, many times over, to be hard problem, not yielding to simplistic
solutions.  

> This is an argument that SUB/NOT should not be used for anything the
No, it's an argument that any attempt to define a congestion control
mechanism that reflects congestion state backwards is hard.  It's not
SUB/NOT that is the problem, it is backwards flowing congestion advice.
Strictly speaking, it's not the advice that is hard, it is describing when
to send it, and how to react to it that is hard.

> TCP would not have throttling mechanisms and offered traffic would be
Of course.  Same can be said of BGP.  An ENORMOUS amount of work has gone
into developing the algorithms that are involved in congestion control in
those protocols.  They are not at all simple, and we didn't arrive at them
whole cloth.  I believe congestion control efforts in SIP will follow a
similar path, and they will involve a bunch of simulation and/or real world
testing.  When we have some, then it will be time for protocol development.

And, I don't see the need yet.  Overengineer the peering system for now,
until we have a more efficient answer.  Even then, it may not be worth
deploying a more complex system.  A lot of the Internet relies on the
realization that more bandwidth (and routing capacity) is cheaper than
complex mechanisms that limit the amount of capacity you need.  That's where
we are right now; cheaper and easier to give you more bandwidth that to
define and implement working congestion control mechanisms.

Brian

---

From: Tim Dwight <[email-masked]>
Date: Wed, 16 Aug 2006 16:17:35 -0500

Mike,

> point.  Are we still addressing the same problem?

I take it that your concern is not to be with congestion *in the fabric",
but rather congestion in one peering partner's network.  If I understand you
correctly you are arguing that means ought to exist whereby the congested
network can notify its peers to reduce the load they are imposing on the
congested network.

If that's not the gist please accept my apologies.  This discussion has been
hard to follow.

If it is, I'd observe that:

1. the problem seems realistic (congestion happens even in "well engineered
networks").

2. solutions to this problem do exist in other domains, so not addressing it
would 
   seem a step backward

3. ideally the semantics for communicating congestion would support both
selective and general
   overload controls.

4. selective controls need to be based on externally-meaningful information.
Given that 
   networks don't typically share topological information this probably
means basing
   such controls on AoRs.  To be scalable we'd need these to aggregate
somehow (so that 
   network A could refer to "all AoRs associated with congested device A.x"
without listing
   them).

tim