Subject: [multipathtcp] Proposed changes to the protocol draft

From: Costin Raiciu <[email-masked]>
Date: Thu, 8 Sep 2011 22:40:14 +0300

Hi all,

As you know, we are in the stage of finalizing the MPTCP protocol draft. There are two major changes still to be made. Having re-read recently all the emails that have gone back and forth, here are the resolutions I propose.

The first question relates to the ability of opening new subflows in fallback mode (break before make in fallback is a specialization of this case). 

My inclination would be to mandate the following: 

- when MPTCP goes into fallback mode, it effectively becomes equivalent to a regular TCP session; no further subflows can be setup; teardown happens only with regular FIN messages, etc.

This does prevent mobility in certain cases that would require a lot of complexity to handle; it seems reasonable given my hope that payload changing middleboxes are not that ubiquitous.

The second question regards adding support for transparent proxies. In particular, each endpoint should know, for each address of their peer, if it is talking to a proxy or the destination. This would allow a smooth transition from
an initial proxy based deployment in cellular networks to worldwide support for MPTCP in servers. In particular, MPTCP could connect to the proxy 

The proposal is to add the P flag to ADD_ADDRESS, to allow middleboxes to say they are on path. 

Currently ADD_ADDRESS does not have flag space; 
- we could add another octet for flag space (but this is not ideal since this option is already big enough for IPv6, as Georg pointed out)
- or we could reduce the number of bits in the address ID by one. 256 addresses seem like a lot of interfaces to have at the same time; 128 should suffice. We should explicitly allow the number to wrap, though, to support long term mobility where a mobile's Wifi interface comes up and down hundreds of times. If we do this, we also free one bit on the MP_JOIN.

My inclination is to go for the second option.

When I first thought of it, it seemed to me that adding the P flag only to add_address is not sufficient; the host does not know upon directly opening a connection/subflow whether it is talking directly to the destination or a proxy. 
It turns out that using add address the proxy can make itself known for the initial path. This path will have address ID 0; the proxy will just send an add address with ID 0 and it's real address. The receiver now compares this address with what he thinks the remote endpoint is, and can infer it is talking to a proxy on that path. The same applies if a new subflow is opened.

What do people think? Are these reasonable solutions? If people agree, I will go ahead and rev the draft accordingly.

Cheers,
Costin
> 
> On 6 Sep 2011, at 18:31, Alan Ford wrote:
> 
>> Folks,
>> 
>> Does anybody agree/disagree with my comments below? Can anybody contribute text for (2) and (5)? These are the main ones that I don't know enough about what was discussed and was intended in order to write anything myself.
>> 
>> Thanks,
>> Alan
>> 
>> On 22 August 2011 22:08, Alan Ford <[email-masked]> wrote:
>> Apologies for the delay in replying.
>> 
>> (1) - This won't work, unfortunately - we don't have enough option space.
>> 
>> (2) - I'm afraid I'm out of touch on what the conclusion here was. When I last thought about this, the only way I could see it working was if a subflow was clearly terminated before a new one was started, relying on the timeout you talk about in (3) to keep the MPTCP state alive. Was there an alternative way?
>> 
>> (3) - Indeed this is heuristics, although more so than for regular TCP applications may make assumptions about session survivability. This is one of those issues that we need to say needs thought in the protocol draft, and then in the API draft talk about its actual impact.
>> 
>> (4) - ok
>> 
>> (5) - Once again I'm out of touch of what was discussed at Quebec so am not sure how this would be defined. I remain uneasy about this flag - embedding proxies as a fundamental part of the protocol - but am willing to bow to consensus at Quebec.
>> 
>> Cheers,
>> Alan
>> 
>> 
>> On 19 August 2011 08:15, <[email-masked]> wrote:
>> Hi,
>> 
>> Just a reminder, please can we try and close these issues
>> 
>> (1)    no-one complained at Mark’s suggestion
>> 
>> (2)    Needs some slightly revised text
>> 
>> (3)    May need some tweaked text?
>> 
>> (4)    Needs no action
>> 
>> (5)    Is the main point requiring a detailed proposal
>> 
>> Thanks
>> 
>> phil
>> 
>> 
>> 
>> From: Eardley,PL,Philip,DES8 R 
>> Sent: 28 July 2011 00:01
>> To: Eardley,PL,Philip,DES8 R; [email-masked]
>> Subject: Open /closed issues from WG meeting
>> 
>> 
>> 
>> From the WG meeting I think we have a proposal on various Issues that seemed open before the meeting.
>> 
>> This email is to check consensus, especially amongst the WG people who aren't in Quebec or didn't listen in.
>> 
>> 
>> 
>> Of the issues, (1) was raised in the meeting, (2) & (3) are from Mark's slide 12, (4) and (5) are from Georg's slide 34.
>> 
>> 
>> 
>> (1) Ack of MP_CAPABLE.
>> 
>> See Mark's email to the list a short while ago, which explains & proposes a resolution.
>> 
>> 
>> 
>> (2) Mobility in fallback
>> 
>> The doc needs to correctly describe the case - when it works OK & when it doesn't (ie what the limitation is). We accept the limitation (it isn't too clear how to do a solution but seems it would be complex)
>> 
>> 
>> 
>> (3) Teardown of state when all subflows fail
>> 
>> This is a heuristics issue rather than a protocol issue.
>> 
>> (Hopefully in the future we can write a doc with lessons learnt from experiments, with suggestions how to deal with all sorts of heuristics issues)
>> 
>> 
>> 
>> (4)  Add Bulk_transfer_optimisation flag to MP-Capable?
>> 
>> Don't add, seems like extra complexity for not much gain
>> 
>> 
>> 
>> (5) Add support for transparent proxy?
>> 
>> Rough consensus in the room in Quebec to add an "I'm a proxy" flag to ADD_ADDRESS.
>> 
>> Need the precise text to add to the draft.
>> 
>> 
>> 
>> We'd like to resolve all these issues within the next week or so - hoping that proves possible!
>> 
>> 
>> 
>> We will WG last call as soon as there's an updated draft that deals with these issues.
>> 
>> 
>> 
>> If I got something wrong, please shout.
>> 
>> 
>> 
>> thanks to everyone for the efforts to try and resolve the hopefully last remaining issues with the protocol spec.
>> 
>> Best wishes,
>> 
>> phil
>> 
>> From: [email-masked] [[email-masked]] On Behalf Of [email-masked] [[email-masked]]
>> Sent: 27 July 2011 23:34
>> To: [email-masked]
>> Subject: [multipathtcp] Draft minutes of WG meeting
>> 
>> thanks to Andrew for taking notes.
>> 
>> 
>> 
>> Here are the draft minutes of the meeting, please send corrections /improvements
>> 
>> thanks
>> 
>> phil/
>> 
>> 
>> 
>> Minutes of MPTCP WG Meeting
>> 
>> WEDNESDAY, July 27, 2011
>> 0900-1130  Morning Session I
>> 
>> Main aim:
>> Conclude on all protocol issues. So that we can WG last call protocol (& API) docs immediately after IETF.
>> 
>> 
>> 
>> 1. Agenda bashing, note well, Milestones – Chairs
>> 
>> 
>> 
>> 2. API update – Michael Scharf
>> 
>> Doc ready for last call, no comments.
>> 
>> 
>> 
>> 3. Congestion control update – Mark Handley
>> Done. Minor comments from IESG and David Black have been resolved.
>> 
>> 
>> 
>> 4. Protocol update & discussion of open issues – Mark Handley
>> 
>> Reviewers could review section 2 (overview)
>> 
>> Small changes to protocol
>> 
>> 
>> 
>> Discussion over sending key-A on the third Ack; there is an issue over the reliable delivery of the key.
>> 
>> Jana: B could echo back the options.
>> 
>> Mark: The issue is how to trigger stopping option sends.  We will double-check if the issue is covered and resolve if required.
>> 
>> 
>> 
>> Discussion on mobility in fallback mode
>> 
>> It doesn't seem that it is possible to reliably use mobility in fallback mode.
>> 
>> Andrew: Isn't that OK?  Can't you just say, if the ACKs haven't arrived, oh well, RST it.
>> 
>> Georg: The draft says that there's no more multipath signalling in fallback.  So you can't get here.
>> 
>> Mark: So, the only resolution seems to be that if you end up with un-ACKed data, you lose.
>> 
>> Costin: You have signalling in the SYNs
>> 
>> Mark: Which gives you a way out, if that is possible.
>> 
>> Georg: I have a proposal for how to cover this later.
>> 
>> Mark: It probably isn't worth trying to optimise this case.
>> 
>> What to do if all subflows are in time out?
>> 
>> Phil: You're saying that is an implementation specific heuristic?
>> 
>> Andrew: What say no interfaces are up?
>> 
>> Mark: Yes, doesn't need specified.
>> 
>> 
>> Phil: Plan is, get these three issues resolved on the list, give it a week or so, and WGLC it.
>> 
>> 
>> 
>> 
>> Experiments on "Is MPTCP deployable?"
>> 
>> Georg: You can't name service providers or enterprises, to what extent are the harmful boxes installed in providers vs enterprises?
>> 
>> Mark: Cellular providers have more, but no monopoly.  We can't claim this to be completely representative.
>> 
>> 
>> 5. (new draft) Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks – Georg Hampel
>> 
>> Mark: Best path probably has lowest energy per byte
>> 
>> Georg: If you have money or energy to burn, you might not care
>> 
>> Mark: Proxy.  One part of me is revolted by supporting transparent proxies explicitly, and we should support them.  Calling the flag I_AM_A_PROXY is more explicit.
>> 
>> Andrew: I agree there are always proxy devices "there"
>> Andrew: ... there are good features here (is it important to tag this as "min" complexity, images are getting big for this part of the stack)
>> Georg: How about recv buffer reduction?
>> Andrew: good idea
>> 
>> Mark: I'm not totally convinced that buffer reduction is real.
>> 
>> Costin: Receive buffer reduction is orthogonal to implementation complexity
>> 
>> Mark: The other question, if you're in path selection mode, do you get the right behaviour over mobile networks.  Assert: you should be able to tell from signal strength etc. if it is a good link... not really.  I can't easily predict from wireless if it's decent, I have to actually try it.
>> 
>> Georg: The signal is not necessarily indicative, could have other variables
>> 
>> Andrew: Wifi links have to train up, which can take a few hundred packets
>> 
>> David: I have in mind a satellite PEP setup, where RTT could be radically different.
>> 
>> Mark: Our intuition is that you should only use one path most of the time, but you want to transition make-before-break and do a soft transition.  You need to get the links adequately probed.
>> 
>> Georg: Both of our points are based on speculation, we need to know how well you do if you shorten the overlap
>> 
>> Costin: that paper assumed there is an oracle that knows the best path that chooses the best path yes: 15% compared to an optimal path selection algorithm that can't be achieved in practice
>> 
>> Costin: you can also have redundant data if you want to probe and avoid hol blocking
>> 
>> Mark: The spec does not prohibit you sending data on multiple paths
>> 
>> Phil: We've had quite a lot of discussion now on heuristic stuff, how you might make this work in practice.  It would be nice to have more experiments.  My impression is that BTO doesn't seem to gain very much
>> 
>> Andrew: Suggest not doing bulk transfer optimisation, it could be really expensive for little gain.
>> 
>> Andrew: Very much in favour of proxy flags, they're everywhere
>> 
>> Mark: I am in favour of adding a bit to support transparent proxies, 1) address of proxy 2) remote supports mptcp - could decide based on this
>> 
>> Andrew: It can be a content inspecting firewall that just wants to do its job
>> 
>> Georg: Proxy is not optimisation, it's a feature
>> 
>> Michio: No such middle box
>> 
>> Mark: If we can preserve the benefits in the presence of boxes that want to be on path, then we should.  If we don't want the content inspected, we need to encrypt.
>> 
>> Georg: 3G providers still get to bill it.
>> 
>> Gorry: There are proxies that need to inform L2 what is going on.
>> 
>> Andrew: Could have network architectures that wouldn't otherwise work that way.
>> 
>> Mark: We should probably try to address this before we move the protocol forward.
>> 
>> Phil: Would like to see people get together in the remainder of the week.
>> 
>> Andrew: I think the bottom three lines of the slide are OK, up to naming.
>> 
>> Costin: +1
>> 
>> 
>> 
>> 6. Call for news on Implementations
>> 
>> Linux version being updated as http://mptcpinfo.ucl.ac.be, also for Nokia N900 & Android Nexus-one
>> 
>> Tim: Implementation is not SMP capable, does anyone know if it is fixed?
>> 
>> <apparently not, docs still say no SMP>
>> 
>> Georg Hampel - in progress on 'lightweight' implementation
>> 
>> Sebastien Roy: Oracle are working on a prototype in Solaris.  Don't know if it can be released yet.
>> 
>> 
>> 
>> 7. Discussion on future of WG - Chairs
>> 
>> 
>> 
>> Discussion last time (roughly) concluded that we pause for experiments & usage: WG hears about implementations, heuristics etc; hears about investigations on possible extensions; not do any protocol standards work. Re-charter in ~ 1year when understand better what the needs are & where the energy is
>> No comments - Chairs will work with AD on re-chartering text, when the current charter is done.
>> 
>> 
>

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 8 Sep 2011 15:46:05 -0500

Hi Costin, all,

First some comments on "opening new subflows in fallback mode":

In case checksums are used, I agree with your inclination to restrict fallback mode to regular TCP. If not checksums are used, however, there are no payload-rewriting middleboxes (by definition). Under such circumstances, DSS with infinity setting could be interpreted as "use this mapping until updated via a new DSS". The next DSS can be sent on any packet and on any subflow.

I strongly advocate such "temporary fallback solution in absence of checksums" for the following reasons:

- There is currently little information about the likelihood to encounter a payload-rewriting middlebox. Such middleboxes may be very rare in fact. It is further possible that such middleboxes also exhibit other MPTCP-obstructing behavior, i.e. swallow of TCP options etc, which obsoletes the problem.

- According to Andrew McGregor, checksum computation stalls hardware-accelerated stack implementations. In an offline discussion, he recommended to avoid them.

- Single-path connectivity is still the typical scenario for wireless settings. MPTCP provides an opportunity in case a second path shows up. The corresponding opportunity cost, however, should be held at minimum. With 20-28B, DSS has an overhead comparable to a TCP-in-TCP tunnel, and it has to be carried over the air interface (a lot for VoIP).

- In my opinion, bulk optimization adds unnecessary delay and processing overhead.

Here are some comments on your ideas about the transparent proxy. If I understand you right, you offer 3 potential solutions:

a) Add a byte to ADD_ADDRESS for the P flag.

b) Use one bit of the address-id for the P flag.

c) Use address-id = 0 in ADD_ADDRESS (or MP_JOIN), which is usually the default for the initial subflow.

I have a preference for solution c). Some issues:
- Q: What happens to the address-id of the interface used for the initial subflow (whose value was 0)? A: This address cannot be used anymore. The initial subflow, however, does not have to be torn down. Once torn down, it cannot be re-established since the address is gone. Is there any problem here?
- Q: How to distinguish "do join" from "I'm a proxy"? A: Address-id = 0 means "I'm a proxy". Address > 0 means "do join". Is that right?

Thanks,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Costin Raiciu
Sent: Thursday, September 08, 2011 3:40 PM
To: Alan Ford; philip.eardley Eardley; [email-masked]; Yoshifumi Nishida; [email-masked] List
Subject: [multipathtcp] Proposed changes to the protocol draft

Hi all,

Cheers,
Costin
>> (1)    no-one complained at Mark's suggestion
>>
>> 1. Agenda bashing, note well, Milestones - Chairs
>>
>>
>>
>> 2. API update - Michael Scharf
>>
>> 3. Congestion control update - Mark Handley
>> 4. Protocol update & discussion of open issues - Mark Handley
>>
>> 5. (new draft) Enhancements to Improve the Applicability of Multipath TCP to Wireless Access Networks - Georg Hampel
>>

_______________________________________________
multipathtcp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 9 Sep 2011 11:02:13 +0300

Hi George,

On Thursday 08 September 2011 wrote Hampel, K Georg (K Georg):
> In case checksums are used, I agree with your inclination to restrict
> fallback mode to regular TCP. If not checksums are used, however, there
> are no payload-rewriting middleboxes (by definition). Under such
> circumstances, DSS with infinity setting could be interpreted as "use this
> mapping until updated via a new DSS". The next DSS can be sent on any
> packet and on any subflow.

If dss-checksums are not used, how can we (by definition) be sure that there 
are no payload-rewriting middleboxes?

If we are in infinity-setting and thus no DSS-checksums are used,  and a 
payload-rewriting middlebox is adding a byte to the byte-stream. After 
changing the subflow, the TCP-layer will offer a corrupted byte-stream to the 
application. Because the normalizing DSS will be one byte less than what we 
already offered to the application. Thus, the first byte on the new subflow 
will not be passed to the application and thus this byte got lost.
I think, we should at all means, avoid a corrupted byte-stream. And we can 
never be sure that this is not going to happen.

As far as I can see, mobility in fallback-mode can only work if we can 
correctly close a subflow before opening the new one.
But how can we correctly close a subflow in mobility, as we do not know, when 
we will loose the access-point?

> - According to Andrew McGregor, checksum computation stalls
> hardware-accelerated stack implementations. In an offline discussion, he
> recommended to avoid them.

The overhead of DSS-checksum calculation is not that much. I only see a 
difference at very high speeds (about 4Gbps).

Cheers,
Christoph

--
Christoph Paasch
PhD Student

IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp
Université Catholique de Louvain

www.rollerbulls.be
--

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 9 Sep 2011 11:14:50 +0300

Hi,

> I strongly advocate such "temporary fallback solution in absence of checksums" 

If checksums are not used, MPTCP should not go into fallback mode - failed checksums were the only ways MPTCP would go into fallback mode. 
Since fallback cannot really happen (or will happen very rarely, if the spec is updated), there is no point in allowing  mobility in fallback mode. 

Fallback mode was designed as a way to signal to endpoints running MPTCP to revert to TCP from a certain sequence number onwards, in the middle of the connection. We expect this to be very rare in practice, so there is no need to add unnecessary complexity for this fallback mode.

MPTCP can revert to TCP in the beginning of the connection also; after this it never comes back to MPTCP mode. Why should we allow MPTCP to revert back from TCP in fallback mode (i.e. mid connection downgrade), and not allow it to revert back in some cases when the connection has been downgraded in the beginning (for instance because options did not get thourgh on data segments)?

Adding complexity to an already pretty complex protocol is probably the wrong thing to do. I think we should keep things simple and mandate that: 

"Once MPTCP reverts to TCP, it MUST NOT revert back to MPTCP aftewards". 

> - Q: How to distinguish "do join" from "I'm a proxy"? A: Address-id = 0 means "I'm a proxy". Address > 0 means "do join". Is that right?

Actually, the only usable solutions in my mind are a) and b)
They would allow a proxy to send an add_address with a new address_id and a proxy flag, telling the endpoint that the advertised address belongs to a proxy.

However, this by itself does not allow the endpoint to tell if it is talking to a proxy on its initial subflow, or any other subflows for which is has not received information via ADD_ADDRESS with the P flag set and the corresponding address ID.

I was using solution (c) for this exact purpose. Once an address is first signalled implicitly (via subflow creation with MP_CAPABLE (address_id = 0) or MP_JOIN (address_id = x)), the proxy can also send an ADD_ADDRESS with the same address_id (0 or x), thus informing the endpoint that is a proxy address.

We thus need to change the rules on accepting ADD_ADDRESS options: when duplicate options are received for the same address ID, they should be processed nevertheless. This shouldn't introduce new vulnerabilities because subflow setup is the thing that matters and that uses crypto.

Cheers,
Costin

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 9 Sep 2011 03:15:10 -0700

Hi Costin,

> What do people think? Are these reasonable solutions? If people agree, I will go ahead and rev the draft accordingly.

I tend to prefer the second option, but you don't want to use another
1 or 2 bits for future purpose?
If we should explicitly allow the number to wrap, it might not be a
big problem for having smaller ID space.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Fri, 9 Sep 2011 08:57:56 -0500

Hi Christoph, Costin, Yoshifumi, all,

Fallback issue: I agree that when checksums are not used, there is no way for the end hosts to tell if payload has been rewritten, and hence, there's no reason to enter fallback due to payload rewriting. Obviously, if payload rewriting still occurs, the connection may get corrupted or it may break.

I'm striving for something else: If checksums are *not* used and only one subflow runs traffic, DSS can principally be omitted. This reduces overhead and improves processing speed. Omission of DSS can be announced via DSS with infinity setting. DSS must be re-inserted as soon as another subflow carries traffic. This is done via conventional DSSs. 

What problems does this simplification raise?

Proxy issue: 
First a clarification (in my opinion): In presence of a proxy, there can *only* *ever* be two independent "end points" that do MPTCP signaling. If host A and host B do MPTCP, the proxy must remain out of the picture and let all MPTCP signaling pass. If the proxy wants to be in the picture, it has to shun host A's MPTCP signaling from host B and respond on behalf of host A.

Let's assume the proxy is in the picture. In this case, the first subflow runs between host A's and host B's IP addresses with address-id=0 assigned to both addresses. Host B's address *cannot* be used for any further subflow. If the first subflow gets torn down, it cannot be reestablished within the same connection. Therefore, apart from supporting the first subflow, host A has *no* use for Host-B's address. If the proxy forwarded to host A its own address with address-id=0, it would clearly mark that (1) its own address should replace address-id=0 and (2) host B's IP address cannot be used for any further subflows. No need for any further flag or byte.

Am I missing something? Thanks.

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Friday, September 09, 2011 6:15 AM
To: [email-masked] List
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Hi Costin,

Thanks,

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 9 Sep 2011 16:29:57 -0700

Hi Georg,

> What problems does this simplification raise?

Please correct me if I'm wrong.
In my understanding, there seems to be two distinct modes in this.

   fallback mode:  where we want to omit MPTCP signaling since it
might not be very safe.
   one path mode: where we want to suppress MPTCP signaling to reduce
the overhead and improve the performance.

I'm so far inclined to disable all MPTCP features (including opening
subflows) in the fallback mode.
For one path mode, I'm thinking that it's possible to implement it
without violating the current spec.
So, I think that it's quite fine to have this mode in implementations.

> First a clarification (in my opinion): In presence of a proxy, there can *only* *ever* be two independent "end points" that do MPTCP signaling. If host A and host B do MPTCP, the proxy must remain…

Well, I'm not sure if this is a good example, But, a mobile MPTCP node
in future might want to say something like this...
"I'm going to leave this network and will come back with new address
after for a while. In the meantime, you can send data to this proxy if
you want"

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Olivier Bonaventure <[email-masked]>
Date: Sat, 10 Sep 2011 11:43:10 +0200

Hello,
>> I'm striving for something else: If checksums are *not* used and only one subflow runs traffic, DSS can principally be omitted. 

We'd need to define precisely define when there is a single subflow that
runs traffic.

Does this means that only one flow has been established ? or only one
subflow is active (e.g. previous subflows have timedout) or there
unacknowledged data on only one subflow ? or another definition ?

We need an unambiguous definition that can be easily supported by
implementations

> This reduces overhead and improves processing speed. 

But this increases the implementation complexity and the number of
corner cases that have to be considered by implementors.

>Omission of DSS can be announced via DSS with infinity setting. DSS must be re-inserted as soon as another subflow carries traffic. This is done via conventional DSSs.

Implementations will also need to hanlde the case when :
- at time t0, there is a single subflow and thus dss=infinite
- at time t1, a second subflow is used and thus we send dss
- at time t2, we go back to a single subflow and dss=infinite

I fear that this would add unnecessary complexity and would prefer to
wait for more implementation experience before looking at such
optimisations in the main spec

>    fallback mode:  where we want to omit MPTCP signaling since it

Agree. As soon as we go in fallback mode, we must stop all MPTCP signalling.

> subflows) in the fallback mode.

Agreed

> after for a while. In the meantime, you can send data to this proxy if

Adding proxy to the main spec will add complexity to the protocol.
Wouldn't it be better to :
- reserve a few bits in the main spec to have some space to add the
proxy bits that are being discussed but leave proxy out of the main spec
- start a new draft that works on these proxy issues

MPTCP has already several use cases without proxies and I think that it
is important to stabilise the main MPTCP spec based on implementation
experience before trying to add new features and new complexity.

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: Costin Raiciu <[email-masked]>
Date: Sun, 11 Sep 2011 17:09:20 +0300

Hi,

I agree that single path mode could be made to work if we use infinite mappings and a lot of care to cover the cases when there are two subflows.

But we can cover the single path mode by using really long data sequence mappings - as long as you want them. And that needs to further thought on corner cases, and does not add further complexity to MPTCP.
Sending one option every few RTTs seems like a reasonable overhead to me.

As for the proxies, all I am proposing is to add a bit to ADD_ADDRESS; that seems to cover all the signaling we need for transparent proxies, which for sure will be used in cellular nets if mptcp is deployed on mobiles.

Cheers.
Costin

---

From: Lars Eggert <[email-masked]>
Date: Mon, 12 Sep 2011 09:27:31 +0300

> optimisations in the main spec

This is my feeling at the moment, too. I think it'd be useful to have this written up in an individual ID so that folks can discuss (and maybe try and implement) it, but I think it's too early to put this into the main spec.

Lars

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 12 Sep 2011 08:40:33 -0500

Hi Yoshifumi, Olivier, Costin, Lars, all,

I tend to agree with Olivier and Lars: Transparent proxy as well as "single-path mode" are new features, which extend the present architecture/design. We should add room for extensions to ADD_ADDRESS and leave it up to further IDs, discussions, implementations etc to decide on further details.

On the discussion of these features:

I agree that it makes sense to differentiate between "single-path" mode and "fallback mode". The expression "single-path" mode leads to misunderstandings, however, as Olivier pointed out. In fact, it should be allowed to have multiple subflows up, but only one of them can carry traffic.

Using bulk optimization with large range as proposed by Costin may lead to problems if sudden handover is needed (e.g. sudden break before make).

Working on an implementation right now, I canNOT confirm that "single path" mode is more complex than the present bulk optimization.

I think that proxy issues brought up by Yoshifumi ("use proxy until I'm back") require further discussion. I don't understand the scenario: Where would the proxy ship the data if the host temporarily moves to a different address? Why can't he host send his temporary address instead of the proxy's address?
Thanks.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lars Eggert
Sent: Monday, September 12, 2011 2:28 AM
To: [email-masked]
Cc: [email-masked] List
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Lars

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 13 Sep 2011 02:35:44 -0700

Hi Georg,

> Working on an implementation right now, I canNOT confirm that "single path" mode is more complex than the present bulk optimization.

I cannot confirm, either.
Probably, the important point at this moment is we don't have to
change the main spec for it.
You can proof it through your experiments later and we can discuss it.
But, If you find problems in fallback mode, I think we'll need to
discuss now since it's currently in the spec.

> I think that proxy issues brought up by Yoshifumi ("use proxy until I'm back") require further discussion. I don't understand the scenario: Where would the proxy ship the data if the host temporari…

It was just an example.. I may be wrong, but I was trying to say that
there might be more than 2 nodes in future mptcp signaling exchange.
I thought presuming only 2 nodes in mptcp signaling might not be a
good idea if we are trying to make the spec extensible.
Sorry if this brings confusion.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 13 Sep 2011 08:29:41 -0500

Hi Yoshifumi,

There seems to be the general notion that implementation and measurements have to catch up with the present specs before new features are added.

Both, "single path mode" as well as transparent-proxy support, would be such features.

Single-path mode further relies on the absence of payload-rewriting middle boxes which needs to be confirmed via trials.

Your comments on the transparent proxy confirm that more discussion is needed in this area.

Thanks.

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Tuesday, September 13, 2011 5:36 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked] List
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Hi Georg,

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 16 Sep 2011 00:47:51 -0700

Hello,

It seems that we're reaching the stage to finalize the spec.
Please correct me if I'm wrong, but here's the current status I think..

Constin's proposal 1:
    "when MPTCP goes into fallback mode, it effectively becomes
equivalent to a regular TCP session; no further subflows can be setup;
teardown happens only with regular FIN messages, etc."

 seems to be reaching consensus as we don't see any objections so far.

Constin's proposal 2:
    "add the P flag to ADD_ADDRESS, to allow middleboxes to say they
are on path"

 seems to need more inputs to discuss where to put the flag, although
this looks relatively a minor issue.

In addition, I guess we need to discuss initial handshake. Or, the
authors already got the conclusion about this?

If someone has any other concerns, please point out!

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Fri, 16 Sep 2011 09:52:03 -0500

Yoshifumi, all,

I have a problem with the quick fix of "I'm a proxy"-bit squeezed into ADD_ADDRESS. I propose to keep the ADD_ADDRESS option as in draft-ietf-mptcp-multiaddressed-04.txt and to discuss further extensions over time.

Here are the reasons:

Currently, the ADD_ADDRESS option covers two meanings, namely:
(1) Cache this address together with address id.
(2) Do MP_JOIN from this address right away.

This works as long as the request to do MP_JOIN is the only purpose for ADD_ADDRESS. That's fine for now, but probably not in the future.

With the introduction of the transparent proxy, ADD_ADDRESS obtains at least one additional meaning:
(3) I'm a proxy address.

We still have to discuss what the implications of this statement "I'm a proxy" are. One implication could be: ONLY use this proxy address for future subflows. Are we certain that this is the only meaning? I'm not. (Yoshifumi doesn't seem to be either).

Further, a MPTCP host may want to provide alternative addresses as backup, e.g. of different IP version, WITHOUT requiring a new subflow right away. This is currently not covered.

I think what we ultimately need is a separation between two functions:
(a) Provide a new address together with address id
(b) Assign a specific interpretation or request-for-action to an address id

Function (a) can be covered by ADD_ADDRESS. Function (b) deserves its own EXTENDIBLE option, which allows adding more interpretations/requests over time.

Thanks,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Friday, September 16, 2011 3:48 AM
To: multipathtcp
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Hello,

Constin's proposal 1:

Constin's proposal 2:

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: <[email-masked]>
Date: Fri, 23 Sep 2011 16:08:27 +0100

There seems to be consensus around completing the main spec without trying to solve the proxy issue (since it needs significant more work), but making the spec flexible enough that a 'proxy capability' can be added later. 
Question is how to leave room for this capability. One suggestion was to keep aside one bit of the address-id - does that have consensus? Would any of the spec have to be re-thought?

Thanks
phil

Philip Eardley   
Research and Technology Strategy

This email contains BT information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you. We monitor our email system, and may record your emails.
British Telecommunications plc
Registered office: 81 Newgate Street London EC1A 7AJ
Registered in England no: 1800000

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lars Eggert
Sent: 12 September 2011 07:28
To: [email-masked]
Cc: [email-masked] List
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Lars

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 26 Sep 2011 10:00:37 -0500

Hi Phil, all,

Are we certain that 1 bit is enough to cover all purposes and implications of ADD_ADDRESS? 

Based on recent proxy-related drafts and discussions, ADD_ADDRESS would already have to support (1) "Join using this address" and (2) "I'm a proxy". Since both implications are independent from each other and may occur in unison, 1 bit would not be enough.

A more detailed discussion on proxy functions may reveal additional implications of this option. 

Adding one byte to the option instead of carving out 1 bit should provide sufficient room for proxy and other extensions. 

Thanks.

Regards,

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, September 23, 2011 11:08 AM
To: [email-masked]
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Thanks
phil

Philip Eardley   

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Lars Eggert
Sent: 12 September 2011 07:28
To: [email-masked]
Cc: [email-masked] List
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Lars

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 27 Sep 2011 02:05:41 -0700

Hi Georg, All,

I think we need more inputs or opinions for this.
I'm trying to create a simple summary for the proposals from the
previous discussion.
Please correct me if I'm missing or misunderstanding something.

 1) Add a byte to ADD_ADDRESS
     pros: simple change to the current spec. good extensibility
     cons: need one more byte

 2) Use one bit of the address-id
     pros: simple change to the current spec.
     cons: less address-id space, low extensibility

 3) Use more bits of the address-id
     pros: simple change to the current spec, some extensibility
     cons: much less address-id space,

 4) Keep the current format and develop new option to describe the
address (e.g DESC_ADDR)
     pros: great extensibility. no need to change the current spec.
     cons: might need another mechanism to sync 2 options reliably.

Also, I personally think the following can be an another possibility,
although it might not be a good idea.

 5) Keep the current format and develop new option to specify proxy or
other addresses (e. g ADD_ADDR2)
    pros: great extensibility. no need to change the current spec
    cons: having 2 options for mostly the same feature might be a bit
ugly design.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 27 Sep 2011 15:49:27 +0200

Yoshifumi,
>      cons: less address-id space, low extensibility

Looking at the ADD_ADDRESS format, we might also revise the IPver nibble :

                           1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +---------------+---------------+-------+-------+---------------+
      |     Kind      |     Length    |Subtype| IPVer |  Address ID   |
      +---------------+---------------+-------+-------+---------------+
      |          Address (IPv4 - 4 octets / IPv6 - 16 octets)         |
      +-------------------------------+---------------+---------------+
      |   Port (2 octets, optional)   |
      +-------------------------------+

         Figure 12: Add Address (ADD_ADDR) option (shown for IPv4)

Currently, we use 4 bits to specify either IPv4 or IPv6. This is a bit
expensive for one bit of information. We could replace these 4 bits by
two bits (to have some flexibility in the future, say IPv4=00 and
IPv6=01) and leave two bits reserved for future use.

Olivier

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Tue, 27 Sep 2011 09:36:03 -0500

Hi Yoshifumi, all,

I think you summarized the various options and their respective tradeoffs very well.

I'm not completely certain, however, on what we are trying to accomplish. While the transparent proxy proposed in our I-D demands one additional bit in ADD_ADDRESS, I'm not sure that one bit is enough to cover *all* (present and/or future) meanings of the ADD_ADDRESS option. 

The WG further seems to have agreed that the proxy concept requires more work. I'm not sure we can anticipate the associated signaling changes without having gone through this work. Why should the present WG draft try to preempt this work?

The proxy discussions may have created the feeling that ADD_ADDRESS *principally* lacks sufficient space to cover future implications/actions/descriptions associated with an address. Such space extension should indeed be captured within the present WG draft. However, I think it should not be guided by preemptive guesses on what a proxy may need.

-Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Tuesday, September 27, 2011 5:06 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]
Subject: Re: [multipathtcp] Proposed changes to the protocol draft

Hi Georg, All,

Thanks,
--
Yoshifumi Nishida
[email-masked]