Subject: [multipathtcp] Multipath TCP Address advertisement 4/5 - Priorities

From: Fabien Duchêne <[email-masked]>
Date: Sun, 6 Nov 2016 13:33:56 +0100

Hello,

As we agreed in Berlin during IETF96, here are a few email to discuss
the contributions proposed in
https://datatracker.ietf.org/doc/draft-duchene-mptcp-add-addr/

The backup mode defined in RC6824 only supports an "all-or- nothing"
mode in the usage of
the subflows, where an host might just prefer to use certain subflow 
over others.
To allow an host to inform the receving host about its preference in
terms of subflow usage, we propose to modify the ADD_ADDR option by
adding 3 "priority" bits.
A host receving a priority for a sub MAY use this information in it's
scheduling decisions.

To allow the priority of an already established subflow to be modified,
we propose to modify the MP_PRIO option by adding the 3 priority bits
next to the "B" flag.

For the moment, the priority field is interpreted as an unsigned integer
with the highest numerical value being the most preferred one, but this
is up to discussions, but this point is up to discussion.

Fabien

---

From: Christoph Paasch <[email-masked]>
Date: Sat, 12 Nov 2016 23:51:45 -0800

Hello,

> is up to discussions, but this point is up to discussion.

while I support adding more info to MPTCP to allow a finer-grained control
of the peer's scheduling, I think the priority-bits should be defined in a
more precise way.

I see the this work here as a way to allow for a more deterministic behavior
when an MPTCP-client connects to a MPTCP-server. As of today, when I on my
mobile device connect to a server that is outside of my control, the only
way I can tell the server to schedule traffic in a certain way is by using
the backup-bit. Upon which I can expect the server to not send traffic on
this subflow unless the primary subflow is broken.
This kind of "configuration" is not sufficient for most use-cases.

I see the priorities as a way for the client to tell the server exactly
what kind of scheduling it expects to meet a certain "QoS-requirement".
So, we should clearly specify what each priority means.

If we leave the interpretation of the priority-bits open to the
implementation, hosts still cannot rely on getting a certain service by
the peer when set the priority bits.

Thoughts?

Cheers,
Christoph

---

From: Fabien Duchêne <[email-masked]>
Date: Mon, 14 Nov 2016 02:58:48 +0100

Hello,

Inline,

> So, we should clearly specify what each priority means.
The way I wrote the draft was more like "ok, let's tell the server that, 
if possible,
I'd like to receive most on the trafic on this subflow, then this one, 
then this one".
This is why I wrote

"The priority field MUST be interpreted as an unsigned integer value with the highest
numerical value being the most preferred one."

While I agree that we should be as specific as possible, I wonder how we could be more
explicit about this. Any idea?
If we are too specific, like saying "always schedule on the highest priority subflow,
until the windows is full, then skip to the next one" we are actually writing a strict priority
scheduler, and I was hoping that we could leave the priority open to more creative ways
of handling the priorities.
It's a MAY because I'm not sure that making the respect of the priority a MUST would
be interesting, for backward compatibility, but also because the priorities of one host
could clash with the other's own interest.

> the peer when set the priority bits.
Cheers!

Fabien

---

From: Alan Ford <[email-masked]>
Date: Mon, 14 Nov 2016 06:14:17 +0000

Hi all,

After discussion at IETF97, I think it’s clear that this proposal - the idea of “priority” - means different things to different people. For example:

a) A percentage split - but would that only kick in when one link is full? Or would all traffic always be split?
b) Prioritising subflows, overflowing only when one is full
c) QoS (latency, bandwidth, etc) values
d) etc etc

Personally I think only (b) makes sense at the subflow level, everything else is far too complex to signal in a few bits. Whether it is of use to people in the real world, however, I don’t know, however!

I also think it’s entirely feasible to achieve a lot of the traffic engineering here by using TCP control signals - ACKs and window size - to slow down communications. As a sender you can already control rate based on your local policy, and at the receiving end you could use ACKs, ECN, window, etc.

This proposal needs a much clearer idea of what it’s trying to achieve before we could consider it for merging.

Regards,
Alan

> On 14 Nov 2016, at 01:58, Fabien Duchêne <[email-masked]> wrote:
> 
> Hello,
> 
> Inline,
> 
> The way I wrote the draft was more like "ok, let's tell the server that, if possible,
> I'd like to receive most on the trafic on this subflow, then this one, then this one".
> This is why I wrote
> 
> _______________________________________________
> multipathtcp mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/multipathtcp <https://www.ietf.org/mailman/listinfo/multipathtcp>

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 14 Nov 2016 07:17:43 +0100

Alan,

> I also think it’s entirely feasible to achieve a lot of the traffic
> engineering here by using TCP control signals - ACKs and window size -
> to slow down communications. As a sender you can already control rate
> based on your local policy, and at the receiving end you could use ACKs,
> ECN, window, etc.

Announcing different receive windows on different subflows is not a good 
idea. RFC6824 notes in section 2.4

    With MPTCP, all subflows share the same receive buffer and advertise
    the same receive window.

This is a safe principle that hosts should respect.

Olivier

---

From: Fabien Duchêne <[email-masked]>
Date: Mon, 14 Nov 2016 16:00:02 +0900

Hello,

On 11/14/2016 03:14 PM, Alan Ford wrote:
> Hi all,
>
>
> Personally I think only (b) makes sense at the subflow level, 
> everything else is far too complex to signal in a few bits. Whether it 
> is of use to people in the real world, however, I don’t know, however!
>
This is what the draft is proposing, however I agree that other 
interpretation is possible (a precentage split or a QoS like I said), so 
I could make this clearer in the draft.
Even if by doing that, we are almost creating a priority scheduler, but 
everyone agrees with that that's ok.

Fabien

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 14 Nov 2016 23:19:09 -0800

Hello,

On 14/11/16 - 06:14:17, Alan Ford wrote:
> Hi all,
> 
> After discussion at IETF97, I think it’s clear that this proposal - the
> idea of “priority” - means different things to different people. For
> example:
> 
> Personally I think only (b) makes sense at the subflow level, everything
> else is far too complex to signal in a few bits. Whether it is of use to
> people in the real world, however, I don’t know, however!

I agree with you that signalling priorities in the sense of (b) is not
necessarily useful. Because, at the end it is then still left to the
decision of the implementer on how he translates the priorities into
scheduling.

I think that one missing piece in MPTCP today is the lack of control a
host can exercise on how the peer should schedule its traffic.

The backup-bit only enables seamless handover. But MPTCP's benefits go
beyond that. Especially for applications sending a thin stream, the delay
benefits can be huge (as has been mentioned in the IETF journal article), if
the scheduling takes RTT into account (while minimizing cell usage for power
& cost reasons).

Allowing to signal this will be a bigger exercise, but I believe
that it is necessary if we want longterm to go to a world where a client can
connect to any webserver (of any implementation) and use MPTCP.

> engineering here by using TCP control signals - ACKs and window size - to
> slow down communications. As a sender you can already control rate based
> on your local policy, and at the receiving end you could use ACKs, ECN,
> window, etc.

Ignoring potential issues with changing the window-sizes or using ECN for a
moment. Using such tricks would only work for bulk transfers. Thin streams
won't be able to reduce/increase the amount of traffic on the secondary subflow.

Cheers,
Christoph

> Regards, Alan
> >> Hello, while I support adding more info to MPTCP to allow a
> >> finer-grained control of the peer's scheduling, I think the
> >> priority-bits should be defined in a more precise way.
> >> 
> >> I see the this work here as a way to allow for a more deterministic
> >> behavior when an MPTCP-client connects to a MPTCP-server. As of today,
> >> when I on my mobile device connect to a server that is outside of my
> >> control, the only way I can tell the server to schedule traffic in a
> >> certain way is by using the backup-bit. Upon which I can expect the
> >> server to not send traffic on this subflow unless the primary subflow
> >> is broken.  This kind of "configuration" is not sufficient for most
> >> use-cases.  I see the priorities as a way for the client to tell the
> >> server exactly what kind of scheduling it expects to meet a certain
> >> "QoS-requirement".  So, we should clearly specify what each priority
> >> means.
> > if possible, I'd like to receive most on the trafic on this subflow,
> > then this one, then this one".  This is why I wrote
> > 
> > "The priority field MUST be interpreted as an unsigned integer value
> > with the highest numerical value being the most preferred one."
> > 
> > While I agree that we should be as specific as possible, I wonder how we
> > could be more explicit about this. Any idea?  If we are too specific,
> > like saying "always schedule on the highest priority subflow, until the
> > windows is full, then skip to the next one" we are actually writing a
> > strict priority scheduler, and I was hoping that we could leave the
> > priority open to more creative ways of handling the priorities.  It's a
> > MAY because I'm not sure that making the respect of the priority a MUST
> > would be interesting, for backward compatibility, but also because the
> > priorities of one host could clash with the other's own interest.
> > 
> >> Cheers, Christoph
> > _______________________________________________ multipathtcp mailing
> > list [email-masked] <mailto:[email-masked]>
> > https://www.ietf.org/mailman/listinfo/multipathtcp
> > <https://www.ietf.org/mailman/listinfo/multipathtcp>

> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: <[email-masked]>
Date: Tue, 15 Nov 2016 07:49:45 +0000

Hi,

we are happy with a solution with the characteristics of (b) for the moment, not sure whether other use cases would need something else. Plus we had a lot of discussions on what „full“ would mean spcifically.

Marcus

Von: multipathtcp <[email-masked]> im Auftrag von Alan Ford <[email-masked]>
Datum: Montag, 14. November 2016 um 07:14
An: Fabien Duchêne <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Betreff: Re: [multipathtcp] Multipath TCP Address advertisement 4/5 - Priorities

Hi all,

Regards,
Alan

On 14 Nov 2016, at 01:58, Fabien Duchêne <[email-masked]<mailto:[email-masked]>> wrote:

Hello,

Inline,

Hello,

Thoughts?

Cheers,
Christoph
Cheers!

Fabien

_______________________________________________
multipathtcp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: <[email-masked]>
Date: Tue, 15 Nov 2016 08:20:04 +0000

i wonder if there are any lessons we can learn from ecn/dscp expereinces?
my rough impression is that 
- a single bit is useful
- multiple qos levels haven't been as useful as might have expected. in retrospect, at least some people think it would have been better to define codepoints to have a particular meaning about the service carried, rather than just relative levels. at least if one hopes it will work inter-domain.

phil
________________________________________
From: multipathtcp <[email-masked]> on behalf of Christoph Paasch <[email-masked]>
Sent: 15 November 2016 07:19
To: Alan Ford
Cc: [email-masked]
Subject: Re: [multipathtcp] Multipath TCP Address advertisement 4/5 - Priorities

Hello,

Cheers,
Christoph

---

From: Martyn Russell <[email-masked]>
Date: Tue, 15 Nov 2016 17:34:22 +0900

Hi all,

On 15/11/2016 16:19, Christoph Paasch wrote:

Hello,

Hi all,

a) A percentage split - but would that only kick in when one link is full?
Or would all traffic always be split?

>From my understanding of RFC6824 in it's current form, the priority does depend on different use cases of the subflows, which is why it will mean different things to people using the MP_PRIO API. The spec currently talks about 2 such cases. The default case being simultaneously sending data over all subflows, but goes on to talk about an "all-or-nothing" failover subflows.

I think if these use cases are all we care about then that's fine and sufficient. But the spec does suggest possible line prioritisation by cost too. In that situation, I believe the spec needs improving because often you don't want all or nothing on subflows which has a premium/cost attached to it.

I agree, but I think there will be an appetite for more use cases than the priority is currently sufficient for.

I agree.

--
Regards,
Martyn

-- SHAREDBAND EMAIL DISCLAIMER --
Email Disclaimer: http://sharedband.com/email-disclaimer<http://sharedband.com/email-disclaimer/>

Shared Band Ltd T/A Sharedband
Registered in England No 04861356, VAT No GB822654826
Registered Office: 40 Princes St, Ipswich, Suffolk, IP1 1RJ UK

---

From: Alan Ford <[email-masked]>
Date: Tue, 15 Nov 2016 09:36:48 +0000

Hi Christoph,

> On 15 Nov 2016, at 07:19, Christoph Paasch <[email-masked]> wrote:
> 
> Hello,
> 

So what semantics would you assign to the priorities? Or are you saying we should leave that up to implementors"

“This defines the priority of each subflow with relation to each other; how a receiver of this option translates this into the scheduling is implementation-specific."

Is that useful to you as an implementor, do you feel? For interop, when you don’t know what the far end will do with this information, would it be useful?

Are there any fundamentals we could/should specify?

Regards,
Alan

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 15 Nov 2016 09:42:59 -0800

Hello Alan,

> So what semantics would you assign to the priorities? Or are you saying we should leave that up to implementors"

Just to make sure that there isn't a misunderstanding: I think we need a
richer signaling than priority-bits.

Personally, I would know what kind of semantics I would assign to each bit,
because I control both client and server. But that's the model I believe we
should not target for.

> Is that useful to you as an implementor, do you feel?

This depends. If the implementor has a very specific use-case and client in
mind, it will be useful.
For someone who wants to implement a generic-purpose implementation it will
be hard to decide.

> For interop, when you don’t know what the far end will do with this information, would it be useful?

Yes, that's my point. I don't think it will be useful for interop.

> Are there any fundamentals we could/should specify?

I think there are a few use-cases (there are probably many more and it would
be good to complete the list):

1. Thin-streams that require low latency:

   Here a client would like to tell the server what threshold (in terms of
   delay) should trigger use of the secondary subflow. This threshold could
   be expressed in milliseconds or relative to the RTT of the primary
   subflow.

2. Bulk-transfers that have a certain bitrate requirement:

   Here a host would like to the tell the peer this bitrate. The peer will
   then only use the secondary subflow if it can't satisfy the bitrate on
   the primary subflow.

   Instead of expressing this as a bitrate, expressing it as a "deadline"
   might be good as well. For example when streaming a video over HTTP, each
   chunk has a certain deadline at which point it should reach the receiver
   to avoid rebuffering of the video.

3. Transfers (thin and/or bulk) that just require reliability:

   The backup-bit does its job here in this scenario.

Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 15 Nov 2016 09:43:56 -0800

Hello Markus,

On 15/11/16 - 07:49:45, [email-masked] wrote:
> we are happy with a solution with the characteristics of (b) for the moment, not sure whether other use cases would need something else.

can you give an example of how you would translate the priority-bits into
scheduling decisions?

Thanks,
Christoph

> Plus we had a lot of discussions on what „full“ would mean spcifically.
> 
> Marcus
> 
>