Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts?

From: "Ford, Alan" <[email-masked]>
Date: Tue, 1 Feb 2011 17:42:30 -0000

Hi Bob, all,

Thanks for your comments. You are of course right that MPTCP is not
appropriate for short flows, the problem naturally being the difficulty
in deciding when a flow is long-lived or not.

The choice of these heuristics would be a very implementation-specific
thing, and I'm not sure how much there is we can recommend for the
protocol draft. However, we'll add a little commentary on this in the
next revision (probably recommending waiting for large scale
experimental results, however - as you say, there may well be a numeric
heuristic we can use).

In the API draft, we already make a comment (Section 4.4):

   The enabling of MPTCP, either by application or by system defaults,
   does not necessarily mean that MPTCP will always be used.  Both
   endpoints must support MPTCP, and there must be multiple addresses at
   at least one endpoint, for MPTCP to be used.  Even if those
   requirements are met, however, MPTCP may not be immediately used on a
   connection.  It may make sense for multiple paths to be brought into
   operation only after a given period of time, or if the connection is
   saturated.

However we can probably say a bit more about this. An enhanced API
should indeed allow an application to signal this (and change its mind
during operation); we will make more mention of that in the Extended API
appendix in the next revision. This would be an enhancement to the basic
"MPTCP on/off" decision - the enabling of MPTCP can be on by default but
there needs to be a way of stating that there is no need for additional
subflows at this time. Come to think of it, this could have protocol
implications and may require an additional flag on the initial MPTCP SYN
exchange.

Regards,
Alan

> -----Original Message-----
> From: Bob Briscoe [mailto:[email-masked]]
> Sent: 31 January 2011 17:15
> To: [email-masked]
> Cc: [email-masked]
> Subject: mptcp-multiaddressed: How long before multipath starts?
> 
> Alan Costin, Mark, or ANOther,
> 
> The large majority of TCP flows are very short. If there will only be
> one or two packets to send, there will be no point sending off loads
> of requests to open sub-flows straight after the first 3-way
handshake.
> 
> So I imagined there would be some advice in one of the MPTCP I-Ds on
> how long a MPTCP should wait until it attempts to open sub-flows.
> 
> I can't find this from a quick scan. I would have expected it to be
> in the heuristics section of draft-ietf-mptcp-multiaddressed:
>
<http://tools.ietf.org/html/draft-ietf-mptcp-multiaddressed-02#section-3
.8>
> 
> In general, a TCP stack doesn't know how much data the application
> might eventually want to send. So I guess the basic advice might be
> to wait n_1 RTTs before opening a second subflow, then n_2 RTTs
> before opening a third and so on.
> 
> Has anyone done any research on this to find the best tradeoff (ie
> best values of n_1, n_2, etc) between improved performance/resilience
> and wasting signalling packets? I imagine this heuristic would be
> clocked by RTTs, not real-time or bytes, because that would likely be
> the relevant parameter for the signalling tradeoff.
> 
> Then, in the API draft, there might be some discussion on whether the
> application can tell the transport how much data it intends to send,
> so the heuristics can be less blind.
> 
> Thoughts?
> 
> 
> Bob
> 
> 
> ________________________________________________________________
> Bob Briscoe,                                BT Innovate & Design

---

From: Dominik Kaspar <[email-masked]>
Date: Wed, 2 Feb 2011 10:48:10 +0100

Hi,

My question might be slightly off topic, but this thread on short
flows motivated me to ask it anyway.

Are there any existing models to predict the TCP throughput after a
given elapsed time (e.g., the throughput after 2 seconds, after 5
seconds, etc...)? The models I am aware of seem not to be applicable
because they all estimate a long-term average. I'd be grateful for any
hints and pointers.

Best regards,
Dominik

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 2 Feb 2011 12:10:35 +0200

Hi,

The short flows question is a valid one; we don't have any heuristics yet, but we did some measurements for a recent submission.
We measured file transfer completion times for different file sizes. The setup was very simple: two computer connected with two gigabit links. We compared TCP with UCL-BE's Linux kernel implementation of multipath TCP to understand when it is worth using multipath TCP. These results (and probably a lot more) could be the starting point to define the heuristics Bob is asking about.

Attached is the figure showing the results. 

This is the description with put in the paper.
Should MPTCP be enabled for all TCP connections in a data center? We connected two dual-interface machines to a gigabit switch, and measured the time to setup a connection and transfer a short file. TCP can only use one inter- face; MPTCP can also use the second, but only after the first subflow has negotiated the use of MPTCP and the second subflow has been established. Figure 17 shows that TCP is quicker for files of less than about 150 packets, but much slower thereafter. Closer analysis shows that our MPTCP implementation slowstarts slightly slower than TCP. The largest difference is about two RTTs just as TCP fills the link, and decreases thereafter. This different does not seem fundamen- tal, and is likely a quirk of slowstart in our implementation. As it is, it would be better to delay 5ms until the first subflow leaves slowstart before starting the second.

Cheers,
Costin

On 2 Feb 2011, at 11:48, Dominik Kaspar wrote:

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 03 Feb 2011 23:45:29 +0000

Alan,

Surely this shouldn't be thought of as primarily an API issue? - the 
basic MPTCP implementation should be able to decide when to start 
opening sub-flows, whether or not the app wants to give it more 
information thru an API.

That's why I suggested at least something needs to be said on this in 
the heuristics section of <draft-ietf-mptcp-multiaddressed-02> (section-3).

Let's put some ball-park numbers on this. My intention is NOT to 
start a discussion about whether these heuristics are correct, just 
to show this question will need an answer in a basic implementation 
(without any API).

I think everyone would agree that it wouldn't be worth creating any 
sub-flows if the transfer needed no more than 3 packets. With an 
initial window of 3, a TCP server can get 3 packets to a client in 
2RTTs (counting from when the client sends the initial SYN). Any more 
sub-flows would be redundant (other than for sending copies for resilience).

Accurate data on what proportion of flows on the Internet consist of 
no more than 3 packets is hard to come by. Perhaps 50% perhaps 75%. 
But even if it's only 25%, that's enough to have to address the 
question of whether a basic MPTCP implementation will need some logic 
to decide whether and when it is going to start opening sub-flows.

Indeed, if one can work out a good default behaviour based on a 
tradeoff between the cost of signalling for extra sub-flows and their 
benefit, I cannot see much reason for an app to want to override this 
- not much need for an API.

To kick-off a discussion on a baseline heuristic, if IW=N, I would 
propose we should ignore the cost of extra SYNs and SYN-ACKs along 
the same arguments as Wischik's "Short Messages". Then, as soon as 
the server sees some data buffered for the client (implying the app 
has got a request for data from the client), it would be worth the 
server opening one sub-flow for each N segments it will have to send 
to empty the buffer (limited by the no. of distinct sub-flows that 
are possible).

Incidentally, this would be a neat way of increasing the effective IW 
while spreading out the packets over different RTTs, so they don't 
all arrive in the queue at the client back-to-back.

The earliest a server could request more sub-flows is in parallel to 
its first SYN-ACK. But at that point it has no data from the client 
yet, so it doesn't know what the client wants, nor how big it will 
be. It could optimistically start a sub-flow though. At this point, 
we start to need some research on whether this would be worth it...

Basically, I'm saying that we shouldn't expect implementers to have 
to be researchers. At minimum the results of even the most basic 
research should be documented in the heuristics section, so an 
implementer who isn't a researcher can just do what it says.

Bob

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 4 Feb 2011 13:19:23 -0000

Bob,

On the surface this sounds like a sensible heuristic to use -
essentially, you are saying, do not use MPTCP until there has been
congestion and thus you want to initiate resource pooling.

However, there are a couple of (smallish) issues with this approach. In
the client/server model, it is the server who is in a position to know
its buffer is filling up and thus how many new subflows it wants. But in
many cases (e.g. client addresses behind NATs) only the client is in a
position to create these subflows - and the client won't know about the
amount of data in the server's buffer. (Some intelligence could be
applied, e.g. opening subflows at the client side when the window size
has stayed static for N packets - this is probably good enough for most
cases, although adding more than one extra subflow would have to wait
until both subflow windows have stayed static).

The other issue with this approach is that it does not provide resource
pooling for resilience, only throughput. Therefore, a second heuristic
about time/number of packets elapsed should be applied, to provide
resilience to longer-lived flows.

Most of this behaviour can be controlled by the endpoints, and does not
require any protocol changes. Or are there any cases where we may want a
flag to say "I speak MPTCP, but don't want to use it yet"?

Regards,
Alan

> Sent: 03 February 2011 23:45
> To: Ford, Alan
> Subject: RE: mptcp-multiaddressed: How long before multipath starts?
> 
> Alan,
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 04 Feb 2011 17:53:34 +0000

Alan,

>congestion and thus you want to initiate resource pooling.

I didn't intend to say that, and I don't think I did. I thought I 
said the decision should depend on how many packet's worth of data 
the end-point has to send.

>amount of data in the server's buffer.

Good point.

Then an API at the client *would* be useful... In my first mail I 
said that, in the common case of an HTTP GET, the first data packet 
from the server usually contains an HTTP header that says how many 
octets the object will be. So, as long as the client has a handy 
casio calculator (or even a processor) it can work out how many 
packets that implies, and decide whether to start more sub-flows.

>(Some intelligence could be

Ooh. I would assume it wants to be opening sub-flows long before it 
would have time to measure such heuristics. The idea of MPTCP is 
surely to have all the sub-flows there, and let congestion control 
decide how much to use them.

>resilience to longer-lived flows.

I assume "this approach" is what you *thought* I said (something to 
do with congestion). I believe, what I actually said satisfies your 
concern here about resilience.

>flag to say "I speak MPTCP, but don't want to use it yet"?

I am not saying there is any need for a richer wire protocol here. If 
I speak MPTCP but don't want to open subflows, surely I just don't 
open a sub-flow, or I don't respond when asked to open one.

But a protocol is more than the wire protocol. It also includes the 
algorithms. I'm saying some baseline algo for the decision when to 
open sub-flows needs to be in the multi-address spec. But made clear 
that it's not mandatory and better heuristics would be encouraged.

Otherwise, if most implementers take the easy option, they will open 
all possible sub-flows as soon as possible, and we will have storms 
of redundant control packets due to all the flows on the Internet 
(25%?) that are no more than 3 packets.

For instance, take Wischik's diagram of the files you access when you 
log in to gmail: 
<http://www.cs.ucl.ac.uk/staff/ucacdjw/Research/shortmsg.html>. At 
the time each was a separate HTTP request. Even if some of these were 
within one TCP connection there's may-be 50 TCP connections just to 
login to gmail (I've just accessed CNN.com and opened 20 new TCPs in 
the process).

Each currently requires a 3-way hand-shake before data flows, and 
most are single packet objects - only one is bigger than 3x 1500B 
packets. A naive MPTCP that opened 3 additional sub-flows for each 
object (say) would send 3 redundant SYNs, SYN-ACKs, ACKs, FINs, 
FIN-ACKs then 3 more FINs & FIN-ACKs in the other direction.* That's 
21x50 = 1050 redundant control packets for a fairly normal case.

Even if I'm over-estimating by a factor of 10, just think of all the 
unnecessary interrupt handling, if nothing else.

Bob

* PS. I haven't actually checked whether MPTCP closes sub-flows like 
this - apologies if I am over-egging the pudding.

> ><http://tools.ietf.org/html/draft-ietf-mptcp-multiaddressed-02#section-
>3> >.8>
> > > >

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 4 Feb 2011 22:54:18 +0100

Hi all,

additionally to the control-overhead that Bob is talking about (all the 
additional SYN's,...) for short-lived flows, there is also the overhead for 
the implementation in generating the keys at the connection initiation (even 
if the flow will be only very short), and trying to send SYN's even if the 
peer may ignore these SYN's (due to the application setting some kind of 
socket-option).

I think that the initial handshake should be as close as possible to regular 
TCP (just a token should be exchanged), and later over this initial subflow, 
the control-information for establishing a new subflow should be exchanged.

Have a look at two past mails, I sended to the list:
http://www.ietf.org/mail-archive/web/multipathtcp/current/msg01300.html
http://www.ietf.org/mail-archive/web/multipathtcp/current/msg01301.html

That way, we are sure that both sides are willing to open a new subflow, 
*before* sending any SYN that tries to establish the new  subflow.

If we just ignore the SYN (if we do not want to open a new subflow), this will 
be costly from an implementation point-of-view. Because the sender of the SYN 
is uselessly allocating the memory for the sub-socket, retransmitting it's 
SYN,...

What do you think?

Cheers,
Christoph

On Friday, February 04, 2011 wrote Bob Briscoe:
> > > the heuristics section of <draft-ietf-mptcp-multiaddressed-02>
> >
> >(section-3).
> >
> > > sub-flows would be redundant (other than for sending copies for
> >
> >resilience).
> >
> > > >appropriate for short flows, the problem naturally being the
> >
> >difficulty
> >
> > > >The choice of these heuristics would be a very
> >
> >implementation-specific
> >
> > > >experimental results, however - as you say, there may well be a
> >
> >numeric
> >
> > > >heuristic we can use).
> > > >
> > > >    The enabling of MPTCP, either by application or by system
> >
> >defaults,
> >
> > > >    endpoints must support MPTCP, and there must be multiple
> >
> >addresses at
> >
> > > >    requirements are met, however, MPTCP may not be immediately used
> >
> >on a
> >
> > > >    connection.  It may make sense for multiple paths to be brought
> >
> >into
> >
> > > >    operation only after a given period of time, or if the connection
> >
> >is
> >
> > > >    saturated.
> > > >
> > > >should indeed allow an application to signal this (and change its
> >
> >mind
> >
> > > >during operation); we will make more mention of that in the Extended
> >
> >API
> >
> > > >appendix in the next revision. This would be an enhancement to the
> >
> >basic
> >
> > > >"MPTCP on/off" decision - the enabling of MPTCP can be on by default
> >
> >but
> >
> > > >there needs to be a way of stating that there is no need for
> >
> >additional
> >
> > > >implications and may require an additional flag on the initial MPTCP
> >
> >SYN
> >
> > > >exchange.
> > > >
> > > >Regards,
> > > >Alan
> > > >
> > > > > The large majority of TCP flows are very short. If there will only
> >
> >be
> >
> > > > > one or two packets to send, there will be no point sending off
> >
> >loads
> >
> > > > > So I imagined there would be some advice in one of the MPTCP I-Ds
> >
> >on
> >
> > > > > I can't find this from a quick scan. I would have expected it to
> >
> >be
> >
> > > > > might eventually want to send. So I guess the basic advice might
> >
> >be
> >
> > > > > best values of n_1, n_2, etc) between improved
> >
> >performance/resilience
> >
> > > > > clocked by RTTs, not real-time or bytes, because that would likely
> >
> >be
> >
> > > > > Then, in the API draft, there might be some discussion on whether
> >
> >the
> >
> > > > > application can tell the transport how much data it intends to
> >
> >send,
> >

--
Christoph Paasch
PhD Student

IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp
Université Catholique de Louvain

www.rollerbulls.be
--

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 7 Feb 2011 00:33:27 -0800

Hi,
I personally prefer to initiate another subflow after the preceding
flow enters congestion avoidance. It looks a simple rule and we can
avoid multiple subflows to perform slow-start at the same time.
Or, does it hurt performance?

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/1/31 Bob Briscoe <[email-masked]>:
> The large majority of TCP flows are very short. If there will only be one or
> two packets to send, there will be no point sending off loads of requests to
> open sub-flows straight after the first 3-way handshake.
>
> So I imagined there would be some advice in one of the MPTCP I-Ds on how
> long a MPTCP should wait until it attempts to open sub-flows.
>
> I can't find this from a quick scan. I would have expected it to be in the
> heuristics section of draft-ietf-mptcp-multiaddressed:
> <http://tools.ietf.org/html/draft-ietf-mptcp-multiaddressed-02#section-3.8>
>
> In general, a TCP stack doesn't know how much data the application might
> eventually want to send. So I guess the basic advice might be to wait n_1
> RTTs before opening a second subflow, then n_2 RTTs before opening a third
> and so on.
>
> Has anyone done any research on this to find the best tradeoff (ie best
> values of n_1, n_2, etc) between improved performance/resilience and wasting
> signalling packets? I imagine this heuristic would be clocked by RTTs, not
> real-time or bytes, because that would likely be the relevant parameter for
> the signalling tradeoff.
>
> application can tell the transport how much data it intends to send, so the
> heuristics can be less blind.
>
> Thoughts?
>
>
> Bob
>
>

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 7 Feb 2011 10:09:03 +0100

Hi,

this sounds like a reasonable approach, to avoid creating congestion due to 
several subflows being in slow-start.
However, I see one problem:
After the initial subflow enters congestion avoidance, we try the next one. 
However the SYN's of this new subflow get blocked somewhere on their path. 
Then we will be stuck with only the initial subflow for a long time (until the 
TCP-stack gives up retransmitting the SYN's of the second subflow). And only 
then we will try the third subflow.

Cheers,
Christoph

On Monday, February 07, 2011 wrote Yoshifumi Nishida:
> > The large majority of TCP flows are very short. If there will only be one
> > or two packets to send, there will be no point sending off loads of
> > requests to open sub-flows straight after the first 3-way handshake.
> > 
> > I can't find this from a quick scan. I would have expected it to be in
> > the heuristics section of draft-ietf-mptcp-multiaddressed:
> > RTTs before opening a second subflow, then n_2 RTTs before opening a
> > third and so on.
> > 
> > values of n_1, n_2, etc) between improved performance/resilience and
> > wasting signalling packets? I imagine this heuristic would be clocked by
> > RTTs, not real-time or bytes, because that would likely be the relevant
> > parameter for the signalling tradeoff.
> > 
> > application can tell the transport how much data it intends to send, so
> > the heuristics can be less blind.
> > 
> > Thoughts?
> > 
> > 
> > Bob
> > 
> > 

--
Christoph Paasch
PhD Student

www.rollerbulls.be
--

---

From: Mark Handley <[email-masked]>
Date: Mon, 7 Feb 2011 09:45:39 +0000

The answer, as with many such things, is "it depends".  And as such,
the protocol specification must avoid being prescriptive on this
issue.  As MPTCP is deployed, we'll discover precisely when it does
and when it does not make sense to start additional subflows, and how
many should be started.

There's clearly no point in starting a second subflow if the first
subflow will terminate before the second ever sends data.  So if
either end knows that the flow will end very quickly, a second subflow
should either not be initiated or not accepted.

  - Sometimes the host stack can know (for example, the app used sendfile()).
  - Often the app can know, but unless it's MPTCP-aware, this doesn't help much.
  - Sometimes the sysadmin knows, and can configure MPTCP on or off
for a particular port that exhibits common behaviour.
  - Sometimes the host stack can predict (it measures how often it
turns out to be useful for a particular port).

I'm sure people will profile implementations in different
circumstances and come up with different heuristics.  For example,
MPTCP in a datacentre with a redundant topology will probably want to
open a lot more subflows than MPTCP over WiFi and 3G on a smartphone.
And in some datacenters, we may wish to disable any fancy subflow
authentication mechanisms, because they serve little purpose.  This
changes the cost of establishing more subflows, so changes the answer.

In the end, after much profiling, we may just decide that it's fine to
always send the second SYN right after the first SYN/ACK in many
enviroments.  After all, even with large windows we send one ack every
two data packets, which is clearly wasteful.  It's just not an
overhead we care about optimising out.  Or maybe wait for the first
subflow to exit slowstart.  But that has a significant cost if the
first subflow has a much higher RTT than the second, and if we set up
the first over 3G and the second would use WiFi, the stack could
easily predict that.

So, I really want to avoid being prescriptive on this issue in the
spec.  It's not necessary to standardize this - it doesn't affect
interoperability or the safe operation of the network.  So we can and
should leave it to implementors to make smart choices based on
implementation and deployment experience.

 - Mark

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 07 Feb 2011 19:07:29 +0000

Mark,

I wasn't asking for the draft to be prescriptive. Merely wanting some 
guidelines from what people have found out so far. And some advice on 
what is very unlikely to be useful (e.g. opening more sub-flows when 
the stack knows how much outstanding data there is and that it will 
likely all have been delivered before a new sub-flow is established).

Clearly, from all the emails on this thread so far, there's some 
experience on this already. There's also views on what ought to be 
done, some with, some without empirical evidence.

Therefore, I don't think the draft should say nothing on this issue.

If "it depends", the draft should at least say what it depends on - 
even repeating what your email says in the draft (under "heuristics") 
would be much more useful than nothing.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 07 Feb 2011 19:23:43 +0000

Christoph, Yoshifumi, Costin,

What is the original logic behind this... Why 
should waiting for slow-start to end on one flow 
be relevant to when to start another sub-flow? I 
don't see why (in general) slow-start on one path 
should affect slow start on another.

I would have thought the opposite: that opening 
more sub-flows would be a more friendly way to 
increase the effective initial window than 
increasing it all on one sub-flow. Because it 
spreads out the overshoots over multiple paths 
(assuming underlying physical path diversity) and 
over time (assuming RTTs are different).

Bob

At 09:09 07/02/2011, Christoph Paasch wrote:
> <http://tools.ietf.org/html/draft-ietf-mptcp-multiaddressed-02#section-3.> > 8>
> > >

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 07 Feb 2011 19:33:00 +0000

Costin,

These are the sort of results I would take care with. Otherwise the 
"chattering classes" will start coining "Costin's Law", which 
mandates "Don't start a second sub-flow until after 150 packets" ;)

I don't want to belittle this, as you're the only guy so far who's 
actually got some empirical evidence. Great!

But to handle this sort of advice, I agree with Mark that we need to 
avoid being prescriptive. We should only using empirical evidence 
when we can explain why the results occurred.

Bob

At 10:10 02/02/2011, Costin Raiciu wrote:
>Hi,
>
>The short flows question is a valid one; we don't have any 
>heuristics yet, but we did some measurements for a recent submission.
>We measured file transfer completion times for different file sizes. 
>The setup was very simple: two computer connected with two gigabit 
>links. We compared TCP with UCL-BE's Linux kernel implementation of 
>multipath TCP to understand when it is worth using multipath TCP. 
>These results (and probably a lot more) could be the starting point 
>to define the heuristics Bob is asking about.
>
>Should MPTCP be enabled for all TCP connections in a data center? We 
>connected two dual-interface machines to a gigabit switch, and 
>measured the time to setup a connection and transfer a short file. 
>TCP can only use one inter- face; MPTCP can also use the second, but 
>only after the first subflow has negotiated the use of MPTCP and the 
>second subflow has been established. Figure 17 shows that TCP is 
>quicker for files of less than about 150 packets, but much slower 
>thereafter. Closer analysis shows that our MPTCP implementation 
>slowstarts slightly slower than TCP. The largest difference is about 
>two RTTs just as TCP fills the link, and decreases thereafter. This 
>different does not seem fundamen- tal, and is likely a quirk of 
>slowstart in our implementation. As it is, it would be better to 
>delay 5ms until the first subflow leaves slowstart before starting the second.
>
>Cheers,
>Costin
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 7 Feb 2011 22:16:57 -0800

I'm a bit curious about if this is a part of the standard or an item
for implementation guideline.
I think some people start working on implementation guideline doc...

--
Yoshifumi Nishida
[email-masked]

2011/2/7 Bob Briscoe <[email-masked]>:
> guidelines from what people have found out so far. And some advice on what
> is very unlikely to be useful (e.g. opening more sub-flows when the stack
> knows how much outstanding data there is and that it will likely all have
> been delivered before a new sub-flow is established).
>
> Clearly, from all the emails on this thread so far, there's some experience
> on this already. There's also views on what ought to be done, some with,
> some without empirical evidence.
>
> If "it depends", the draft should at least say what it depends on - even
> repeating what your email says in the draft (under "heuristics") would be
> much more useful than nothing.
>
>
> Bob
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 7 Feb 2011 22:18:14 -0800

Hello Bob,

It's nothing special. My concern is we cannot always presume path
diversity. So, I just prefer conservative approach.
If we can have good knowledge about network configuration like data
center scenarios, I think we won't have to be conservative. But, it
looks a bit special case to me.

Thanks,
--
Yoshifumi Nishida
[email-masked]

2011/2/7 Bob Briscoe <[email-masked]>:
> What is the original logic behind this... Why should waiting for slow-start
> to end on one flow be relevant to when to start another sub-flow? I don't
> see why (in general) slow-start on one path should affect slow start on
> another.
>
> I would have thought the opposite: that opening more sub-flows would be a
> more friendly way to increase the effective initial window than increasing
> it all on one sub-flow. Because it spreads out the overshoots over multiple
> paths (assuming underlying physical path diversity) and over time (assuming
> RTTs are different).
>
>
>
> Bob
>
>> > > one
>> > > how
>> > > might
>> > > best

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 08 Feb 2011 11:13:05 +0000

Yoshifumi,

Fair enough. It's only in special cases that a 
data centre will fiddle around with its OS stack. 
In most data centres they will just use the default stack.

As we're still experimental here anyway, it's 
fine to be conservative, then people who want to 
experiment with more aggressive behaviours can do so.

Cheers

Bob

---

From: "Ford, Alan" <[email-masked]>
Date: Tue, 8 Feb 2011 11:36:44 -0000

Bob, all,

> packets that implies, and decide whether to start more sub-flows.

It would be nice if such things weren't dependent on the application
having to think about this. And it is important that both ends can be in
a position to open MPTCP subflows.

This are the kinds of heuristics that should be discovered over
experimental deployment. The application-specific aspect is also
something worth noting the API draft appendix.

> decide how much to use them.

Are there any alternative ways of inferring that a server has a lot of
data to send?

> concern here about resilience.

Not entirely. What you were saying, as I understand it now (things got
conflated in my brain), is to do with how much data is queued at the
sender. In the case of, say, SSH, Instant Messaging, or other
low-bandwidth, long-lived flows, this threshold may never be hit, but
resilience is of significant interest (indeed, probably more so than for
many of the high bandwidth flows).

It may be appropriate for such apps to use an API - again something for
noting in the API draft. However, I think a long-life heuristic is
necessary and also one of the easiest to specify, monitor and tweak.

> that it's not mandatory and better heuristics would be encouraged.

I think it would be appropriate that we include discussion on this, but
should conclude with a "implementations MAY choose" a simple heuristic
and pointers to look out for further research.

> (25%?) that are no more than 3 packets.

Quite. Nobody wants this.

Regards,
Alan

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 08 Feb 2011 17:10:45 +0000

Alan,

I think we are violently agreeing on all the points. You had one question...

> > >until both subflow windows have stayed static).

Before the question, I have to interject here too...

Window stability is not a good metric IMO; If a long flow passes thru 
a shared bottleneck with other flows coming and going, its window 
will vary, but this does not mean it's not a long flow.

>Are there any alternative ways of inferring that a server has a lot of

So far Michael Sharf has given the best way I can think of; on the 
list earlier in this thread:

At 17:46 31/01/2011, SCHARF, Michael wrote:
>The simpler alternative is to just observe in the TCP stack
>how much data the app actually writes and then to use this information
>as input to the heuristics.

Bob