Subject: Re: [Simple] MSRP congestion control issue when using relays

From: Cullen Jennings <[email-masked]>
Date: Wed, 2 Jan 2008 15:00:46 -0800

There are two models to implement a relay ....

One is the infinite storage model. This is the easiest one to do and  
more or less solves the congestion issues. Some people find the  
infinite storage is available assumption to be very reasonable (I  
think Rohan is in that camp) and some find it to be so totally insane  
it makes our heads explode ( suspect Adam might be in that camp ).

The other model is the pipes model. If you have data coming in on  
pipe A and it needs to be copied to pipe B, then you read very slowly  
off pipe A as long as pipe B is congested and rely on TCP push-back.  
To make this not have HOL blocking problems, you need to buffer at  
least the chunk size and you need to control the kernel buffer sizes.  
You also needs some sane assumptions about the ratio of the slow pipe  
to size the kernel buffers and the amount of time before message from  
pipe C that goes to B can get interrupt the message from A. I imagine  
a nice masters these could be written on this.

My memory is the spec allows either of these models to work but does  
not really describe how to implement  either of them. If all you are  
doing is IM, well infinite storage may work while if you are doing  
file transfer, the pipe model might be more appealing.

Cullen <with my individual hat on>

On Dec 13, 2007, at 12:46 PM, Robert Sparks wrote:

> My memory (after dredging it for a few days) recalls that we ended  
> up with agreement that there were many ways to handle these kinds  
> of problems and that we would develop some experience around them.
> Having one connection between relays was something explicitly  
> agreed to. Relays were expected to have significant, but not  
> infinite storage, and how the relay chose to manage that resource  
> was its own business.
>
> IIRC the question on having a slow outgoing leg back up the  
> incoming legs was put down when we realized the relay can just  
> close that outgoing leg and tell the incoming side of whatever was  
> going down that outgoing leg that the session's over. Other  
> existing sessions coming through those incoming pipes and going  
> down other outgoing pipes won't be affected. (Rohan - you may want  
> to chime in here).
>
> RjS
>
> On Dec 13, 2007, at 2:33 AM, Ruud Klaver wrote:
>
>>
>> On Dec 13, 2007, at 12:07 AM, Paul Kyzivat wrote:
>>
>>> Its been a very long time, but some memories of discussions on  
>>> this subject are finally coming back to me...
>>>
>>> As I recall, there were in fact extensive discussions on this and  
>>> related topics. IIRC we ultimately concluded that we could not  
>>> solve every problem and so agreed to give up on solving this one.  
>>> I think the assumption was that the relays could be assumed to  
>>> have a lot of resources, such as the ability to buffer to disk to  
>>> mitigate this problem.
>>>
>>> I know that's not a very good answer, but its what I remember. I  
>>> may however be remembering wrongly. Hopefully somebody else can  
>>> confirm and fill in details, or else provide a different answer.
>>>
>>> 	Paul
>>
>> Thanks for responding as someone who took part in the original  
>> discussion. As I said, I tried reading up on it in the archives,  
>> but I'm sure I didn't catch all of it.
>>
>> As an implementor I am mostly interested in a practical solution  
>> and getting something that works. Assuming infinite storage does  
>> not sound very practical to me. Using one TCP connection per  
>> session sounds like a much more plausible compromise.
>>
>> Of course I can just go ahead and implement it any way I want, but  
>> I really wanted to raise this issue in the interest of  
>> interoperability, since the RFCs are just unclear about this.
>>
>> Ruud Klaver
>>
>>
>> _______________________________________________
>> Simple mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/simple
>
>
>
> _______________________________________________
> Simple mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/simple

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Ruud Klaver <[email-masked]>
Date: Mon, 7 Jan 2008 11:06:18 +0100

Hi,

Thank you both for the replies.

On Jan 3, 2008, at 12:00 AM, Cullen Jennings wrote:

> One is the infinite storage model. This is the easiest one to do  
> and more or less solves the congestion issues. Some people find the  
> think Rohan is in that camp) and some find it to be so totally  
> insane it makes our heads explode ( suspect Adam might be in that  
> camp ).

I'm afraid that, as an implementor, I am firmly in the explosion camp.

> pipe A and it needs to be copied to pipe B, then you read very  
> slowly off pipe A as long as pipe B is congested and rely on TCP  
> push-back. To make this not have HOL blocking problems, you need to  
> buffer at least the chunk size and you need to control the kernel  
> buffer sizes. You also needs some sane assumptions about the ratio  
> of the slow pipe to size the kernel buffers and the amount of time  
> before message from pipe C that goes to B can get interrupt the  
> message from A. I imagine a nice masters these could be written on  
> this.

You are going too fast for my slow and explosion-prone brain. Are you  
talking about session sharing on these TCP connections or not? If so,  
how will this prevent HOL blocking issues exactly? What do you mean  
by chunk size? If I want to send someone a DVD (not an uncommon  
scenario) I will attempt to send one chunk of 4.4GB.

> My memory is the spec allows either of these models to work but  
> does not really describe how to implement  either of them. If all  
> you are doing is IM, well infinite storage may work while if you  
> are doing file transfer, the pipe model might be more appealing.
>

I don't think there is anything that prevents users from doing file  
transfers over relays that were designed just for IM.

>> down other outgoing pipes won't be affected. (Rohan - you may want  

So a session will receive the death sentence just for being  
significantly slower than the rest? Neat, I want to implement that on  
our roads.

Ruud Klaver

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Tue, 8 Jan 2008 12:16:12 +0200

> it makes our heads explode ( suspect Adam might be in that camp ).

Infinite storage works for small messages. In a sense, that's what XMPP and 
even SIP itself, do. That's also what SMTP does, and may partially account 
for SMTP servers refusing to relay overly large emails.

Unfortunately, where MSRP is envisioned as a file transfer protocol rather 
than only an IM protocol, this fails. It was pointed out to me that XMPP 
in-band file transfer probably has the same problem - lets not blame only 
MSRP.

> a nice masters these could be written on this.

The only way to solve HOL is for the receiver to drop packets coming from 
upstream that would overflow the queue to downstream. Because MSRP has no way 
to signal transient congestion of a single session when multiple sessions are 
multiplexed over the same TCP connection.

> file transfer, the pipe model might be more appealing.

My understanding is that model one works if file transfers are banned, and the 
second model does not work at all because MSRP is missing critical pieces for 
such a model to work (i.e. any way to signal congestion on per MSRP session 
rather than per TCP session).

-- 
Rémi Denis-Courmont

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 8 Jan 2008 14:51:21 -0800

> rather than per TCP session).

I don't think the critical pieces for this are missing. It signals it  
on the TCP session level (using normal TCP push back) and then the  
sender interrupts the current chunk and switches to other chunk. So  
the multiplexing is at the MRSP sessions level but normal TCP  
mechanisms are used to signal the feedback.

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Wed, 9 Jan 2008 09:56:02 +0200

> mechanisms are used to signal the feedback.

When a TCP session backs off between two relays, it would normally mean the 
link between them is saturated.

Of course, it may also be that the downstream relay dequeues too slowly at 
application layer, which is what you seem to refer to. Unfortunately, 
upstream cannot tell the difference, because the typical TCP API does not let 
the application know why the TCP circuit is congested.

But lets assume for a minute that the upstream relay would still know that 
the "push back" means that it should stop sending chunks from the session the 
currently transmitted chunk belongs to. So what? Does it abort that session? 
Otherwise, when does it resume transmission to it?

No matter how you put things. If you try to transmit multiple streams on top 
of the same TCP circuit without per-stream congestion, you will have HOL 
blocking, *or* you will need infinite buffers *and* infinite timeouts.

-- 
Rémi Denis-Courmont

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 9 Jan 2008 07:22:07 -0800

On Jan 8, 2008, at 11:56 PM, Rémi Denis-Courmont wrote:

> > > such a model to work (i.e. any way to signal congestion on per  
> MSRP
> > > session
When the relay node wants to push back it starts reading slowly from  
the TCP session. The sending node has another messages that it wants  
to send on this TCP pipe so after a bit of time it interrupts and  
swaps, then the relay can start reading fast for the new message.  
There is no  head of line blocking but there is a delay time where  
where one message can delay another - however this time is not  
related to how long the first message is. The length of this time is  
related to the slow read rate, and the size of pipe including the TCP  
buffers on either side. This is very analogous to MTU on a normal  
link. A fair amount of thought went into this and is why the chunk  
size designed to be so small.