Subject: Re: [multipathtcp] Number of DSS to store

From: "Ilpo Järvinen" <[email-masked]>
Date: Sat, 30 Jun 2012 22:01:25 +0300 (EEST)

On Sat, 30 Jun 2012, Alan Ford (alanford) wrote:

> I see. Well, as the draft is written at the moment there is no restriction
> on you sending all mapping upfront. Furthermore, there is no restriction on
> the mapping having to fit within the advertised window. So you could, in
> theory, send the mapping for your whole sendfile() upfront. I don't know if
> existing implementations would have a problem with that, though. MPTCP is
> not really intended to be used in this way, however, since doing this
> defeats the point of responding to congestion signals – dynamically
> readjusting which data are sent on which subflow according to
> congestion/throughput.
> 
> I don't know if implementors have a view on this, and whether we need to
> restrict anything in this are?

I can understand his question pretty well... having been there myself and 
wondered exactly the same thing and being surprised how unhelpful the 
mptcp docs were about this (I was doing a review of the Linux 
implementation back then). It's crucial for the receiver side 
implementation to know if the DSS stuff points to the current segment or 
not, ie., whether the sender can send arbitary mapping or no (even though 
it certainly doesn't make much sense), and in what order to expect them.

...My implementor point-of-view conclusion was back then summarized (in 
private discussions) in these points:

"In summary, I think the docs should specify these two requirements with
MUST (but I might be willing to negotitate on the second point if some
good enough point is raised against it):
1) Mapping for subflow seqno x MUST not be sent before the mappings for
subflow seqno 0 .. x-1 are/have been sent.
2) A mapping MUST appear, at latest, in the segment where is the first
byte it covers is."

...The second point turned out to be impossible to guarantee at the 
receiving end because of middlebox behavior but I'd still "SHOULD be 
send, at latest," that in the docs to bound how far middleboxes could
take it even when they try hard.

In order to push data from subflow upwards one needs to locate the 
relevant mapping(s) which is the reason for the above wishlist, 
monotonicity in particular would be pretty helpful from implementation 
pov.

-- 
 i.

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 10:29:05 +0200

Hello,

I agree with Ilpo that this should be simplified.

The DSS-options should be sent on the segments whose mapping they are 
covering.

Thus, if a DSS-mapping goes from x -> y in the subflow-seqno space, the DSS-
option MUST be on one of these segments.

Actually, that's what we currently do in our implementation because it would 
become very difficult to handle if mappings can come at any point. If the 
segment's seqno is not part of the mapping, I reset the subflow.

Cheers,
Christoph

-- 
IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://mptcp.info.ucl.ac.be
Université Catholique de Louvain
--

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 08:55:12 -0500

All,

Mahesh has raised an interesting point.  Mahesh's use case also applies to wireless access environments where only one subflow is active for a long time, e.g. because only one link is up or other links have very low link quality.

Currently, MPTCP supports the "bulk optimization" feature, where the DSS specifies a mapping that applies to future packets. The amount of data mapped in bulk optimization is limited to the max value that can be carried in the 16bit range field, which amounts to ~65kB or ~43 packets Ethernet packets. This is very small.

To better address the above use case, we proposed the "temporary fallback" feature some time ago. This feature allows MPTCP to fallback to conventional TCP operation on the active subflow, i.e. fixed mapping with no further DSS exchange, until a "multipath reinstatement" option is sent.

I think we should continue working on such a feature given the growing interest and the added benefits to MPTCP.

Regards,

Georg 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Paasch
Sent: Monday, July 02, 2012 4:29 AM
To: [email-masked]
Cc: Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Hello,

Cheers,
Christoph

On Saturday 30 June 2012 22:01:25 Ilpo Järvinen wrote:
> > defeats the point of responding to congestion signals - dynamically
-- 
IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://mptcp.info.ucl.ac.be
Université Catholique de Louvain
--
_______________________________________________
multipathtcp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 16:13:26 +0200

Hi Georg,

On Monday 02 July 2012 08:55:12 Hampel, K Georg wrote:
> To better address the above use case, we proposed the "temporary fallback"
> feature some time ago. This feature allows MPTCP to fallback to
> conventional TCP operation on the active subflow, i.e. fixed mapping with
> no further DSS exchange, until a "multipath reinstatement" option is sent.

I remember this discussion. But at the time it was unclear how this can work 
across payload-modifying middleboxes. Do you have a solution to this now?

> I think we should continue working on such a feature given the growing
> interest and the added benefits to MPTCP.

What is the benefit (increased goodput, reduced delay/jitter, ...) of sending 
the DSS-mapping-option separated from the data itself ?

In my opinion this just increases the implementation complexity, as we then 
need to support receiving many DSS-mappings before receiving the actual data. 
And all these DSS-mappings need to be stored somewhere for an unsure amount of 
time...

Cheers,
Christoph

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 09:09:34 -0500

Ilpo, Christoph,

I'd like to better understand your comments:

(1) I think we agree that the DSS mapping for byte x should ideally arrive not later than byte x itself. 

(2) We probably also agree that this is not always possible because of middle boxes that tear apart TCP options and payload, and delay the option.

I don't see why we should prohibit sending mappings preemptively. This would guarantee the availability of mappings when needed, even if they were delayed by middle boxes. 

Obviously, preemptive mappings have to be stored on the receiver side. This, however, uses very little less space and can be done very efficiently. 

Am I missing something here? Thanks.

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Paasch
Sent: Monday, July 02, 2012 4:29 AM
To: [email-masked]
Cc: Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Hello,

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 16:43:07 +0200

On Monday 02 July 2012 09:09:34 Hampel, K Georg wrote:
> I don't see why we should prohibit sending mappings preemptively. This would
> guarantee the availability of mappings when needed, even if they were
> delayed by middle boxes. 

The mapping is only needed after all the data has been received belonging to 
this mapping (we have to wait for all the data, to verify the DSS-csum).

Thus, a mapping covering bytes x->y should be on any of the bytes between x 
and y.

> Obviously, preemptive mappings have to be stored on the receiver side. This,
> however, uses very little less space and can be done very efficiently. 

The question is, how many of these preemptive mappings do you store? And what 
if you reach the limit of the allowed stored mappings? Probably you should 
then drop some of the received mappings. But if you drop them, when the data 
comes in you don't know the mapping and have to drop the data.
Thus, the data-level retransmission timer has to retransmit the data and this 
will kill the performance.

I imagine that these preemptive mappings would be sent on duplicate acks. We 
then have to ensure reliable delivery of these mappings (thus need some 
mechanism to acknowledge this), because otherwise again the data-level 
retransmission timer kicks in and performance drops.

I think it's much cleaner if we prohibit preemptive mappings.

Christoph

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 09:46:58 -0500

Christoph, all,

Yes, temporary fallback can be done. A thorough WG discussion would be appreciated.

Georg

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 10:13 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Hi Georg,

Cheers,
Christoph

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 10:12:19 -0500

Christoph,

I have implemented preemptive mapping. It's trivial. 

You need only 16B per mapping range: SSN (4B), DSN (8B), range (4B). Note that 16B can cover a large contiguous range! It's NOT 16B per DSS!

For severe multipath operation, the sender probably doesn't plan ahead by more than a few mappings. So that's a few times 16B. For single-path operation, there is only one mapping range, i.e. only 16B.

I implemented the mapping as a list to provide fast insertion/deletion. By keeping a pointer to the last lookup, you can typically retrieve information in const time. Insertion is also const time since mostly in back of list. Same for deletion since the front of list is contiguous, i.e. only the front entry is affected.

DSS should not be sent on separate ACKs since the receiver would consider them as congestion signals.

Georg 

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 10:43 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Christoph

---

From: Anumita Biswas <[email-masked]>
Date: Mon, 02 Jul 2012 10:12:48 -0700

One advantage of sending the DSS with the corresponding data or atleast a chunk of the data is that it is reliably sent. The arrival of the ack for the data indicates that the receiver received the DSS option also. Sending DSS options on ACKs should be prohibited as there are no ACKs for ACKs.

What is the use case for sending multiple DSS options each of 64K length ahead of the data itself in cellular/wifi environments? Even if the bdp is large enough to hold more than 64K of data, reacting to loss and falling to an alternate path relies on MPTCP operation to be ON all the time no?

For offloading 64K to a TSO card, one obvious solution seems to be to send the DSS option on the first non offloaded segment, followed by the remaining data offloaded to the NIC. Alternately, if the TSO NIC supports copying the TCP options in all segments of the large packet, like it would with TCP timestamps, one can imagine offloading the entire 64K data to the NIC. The receiver should be able to drop/ignore the duplicate DSS options.

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 20:39:01 +0200

Georg,

but still, what is the benefit of preemptive mapping in terms of goodput, 
delay, robustness,... ?

All these little "features" just increase MPTCP's complexity and thus make it 
harder for other operating systems to implement it. Thus, decreasing MPTCP's 
chance of widespread deployment.

In my opinion, unless there is a real benefit, we should keep MPTCP as simple 
as possible.

Cheers,
Christoph

On Monday 02 July 2012 10:12:19 Hampel, K Georg wrote:
> You need only 16B per mapping range: SSN (4B), DSN (8B), range (4B). Note
> that 16B can cover a large contiguous range! It's NOT 16B per DSS!
> 
> For severe multipath operation, the sender probably doesn't plan ahead by
> more than a few mappings. So that's a few times 16B. For single-path
> operation, there is only one mapping range, i.e. only 16B.
> 
> I implemented the mapping as a list to provide fast insertion/deletion. By
> keeping a pointer to the last lookup, you can typically retrieve
> information in const time. Insertion is also const time since mostly in
> back of list. Same for deletion since the front of list is contiguous, i.e.
> only the front entry is affected.
> 
> > I don't see why we should prohibit sending mappings preemptively. This
> > would guarantee the availability of mappings when needed, even if they
> > were delayed by middle boxes.
> 
> > Obviously, preemptive mappings have to be stored on the receiver side.
> > This, however, uses very little less space and can be done very
> > efficiently.
> The question is, how many of these preemptive mappings do you store? And
> what if you reach the limit of the allowed stored mappings? Probably you
> should then drop some of the received mappings. But if you drop them, when
> the data comes in you don't know the mapping and have to drop the data.
> Thus, the data-level retransmission timer has to retransmit the data and
> this will kill the performance.
>

---

From: Mahesh M <[email-masked]>
Date: Mon, 2 Jul 2012 12:13:20 -0700

If we don't allow  preemptive mappings and making MUST to send mappings only along with first data segment belonging to the mapping is a good idea. It is simple and works in all the cases.

Regards,
Mahesh

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 11:39 AM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Georg,

but still, what is the benefit of preemptive mapping in terms of goodput, delay, robustness,... ?

All these little "features" just increase MPTCP's complexity and thus make it harder for other operating systems to implement it. Thus, decreasing MPTCP's chance of widespread deployment.

In my opinion, unless there is a real benefit, we should keep MPTCP as simple as possible.

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 21:22:10 +0200

On Monday 02 July 2012 12:13:20 Mahesh M wrote:
> If we don't allow  preemptive mappings and making MUST to send mappings only
> along with first data segment belonging to the mapping is a good idea. It
> is simple and works in all the cases.

Guarantee that it will be on the first data-segment is not possible due to 
segment-splitting middleboxes who might copy the TCP-option only on the second 
packet.

But this is not a huge problem.
In our implementation we just store the segments without mapping in the 
subflow-receive-queue until all segments of the mapping have arrived together 
with the required DSS-option. The DSS-mapping is temporarily stored in the 
TCP-sock.
No need to dynamically allocate 16B (actually it would be 32B as this is the 
minimal allocation-size) and store them in a list and thus potentially have 
additional cache-misses and burn more CPU-cycles.

Christoph

---

From: Mahesh M <[email-masked]>
Date: Mon, 2 Jul 2012 12:54:01 -0700

Guarantee that it will be on the first data-segment is not possible due to segment-splitting middleboxes who might copy the TCP-option only on the second packet.
Mahesh> Is there an example of this? Which proxy does this?

Regards,
Mahesh

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 12:22 PM
To: Mahesh M
Cc: Hampel, K Georg (K Georg); [email-masked]; Ilpo Järvinen
Subject: Re: [multipathtcp] Number of DSS to store

On Monday 02 July 2012 12:13:20 Mahesh M wrote:
> If we don't allow  preemptive mappings and making MUST to send 
> mappings only along with first data segment belonging to the mapping 
> is a good idea. It is simple and works in all the cases.

Guarantee that it will be on the first data-segment is not possible due to segment-splitting middleboxes who might copy the TCP-option only on the second packet.

But this is not a huge problem.
In our implementation we just store the segments without mapping in the subflow-receive-queue until all segments of the mapping have arrived together with the required DSS-option. The DSS-mapping is temporarily stored in the TCP-sock.
No need to dynamically allocate 16B (actually it would be 32B as this is the minimal allocation-size) and store them in a list and thus potentially have additional cache-misses and burn more CPU-cycles.

Christoph

--
IP Networking Lab --- http://inl.info.ucl.ac.be MultiPath TCP in the Linux Kernel --- http://mptcp.info.ucl.ac.be Université Catholique de Louvain
--

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 21:59:01 +0200

On Monday 02 July 2012 12:54:01 Mahesh M wrote:
>> segment-splitting middleboxes who might copy the TCP-option only on the
>> second packet. 

Actually, I don't know of an example of such kind of middlebox.

In Michio's paper "Is it still possible to extend TCP?" they did not found 
such kind of a middlebox.
However, their sample-size is limited and who knows if there exists such kind 
of a (maybe buggy) middlebox somewhere on the Internet.

MPTCP should also pass by such kind of a middlebox.

Cheers,
Christoph

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:20:50 -0500

For our packet-filter proxy implementation DSS delay is a huge problem since we don't buffer packets but only Seq No mappings. If the mapping is not available when the packet arrives the packet has to be dropped. 

The mapping may cost us 16B but we save 1500B for every packet that's covered by this mapping.

We ran our implementation on LTE carriers and big ISPs in the US and didn't account any problem. This means that segment-splitting middleboxes don't seem to be that frequent.

I also wonder if payload-rewriting middleboxes really exist...

Georg  

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 3:22 PM
To: Mahesh M
Cc: Hampel, K Georg (K Georg); [email-masked]; Ilpo Järvinen
Subject: Re: [multipathtcp] Number of DSS to store

Christoph

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Mon, 2 Jul 2012 15:31:06 -0500

Christoph,

For packet-filter implementations bulk optimization and temporary fall back can leverage hardware acceleration or other forms of local processing since only IPs, ports, Seq No and Ack No have to be rewritten.

This is a huge benefit for MPTCP proxies which are instrumental to MPTCP's deployment.

Georg

-----Original Message-----
From: Christoph Paasch [mailto:[email-masked]] 
Sent: Monday, July 02, 2012 2:39 PM
To: Hampel, K Georg (K Georg)
Cc: [email-masked]; Ilpo Järvinen; Mahesh M
Subject: Re: [multipathtcp] Number of DSS to store

Georg,

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 02 Jul 2012 22:37:43 +0200

On Monday 02 July 2012 15:20:50 Hampel, K Georg wrote:
> We ran our implementation on LTE carriers and big ISPs in the US and didn't
> account any problem. This means that segment-splitting middleboxes don't
> seem to be that frequent.

If they are not so frequent, putting the DSS-option on the first packet that 
is covered by the mapping should be sufficient for you. No need to send them 
preemptively.

> I also wonder if payload-rewriting middleboxes really exist...

FTP across NAT. The Linux Kernel implementation does modify IP-addresses if 
FTP is used across NAT, and the ftp-helper module is loaded in the kernel.

I took a quick look at the source-code and it does not seem to me that they 
remove TCP-options. Thus, MPTCP needs to deal with this.

Christoph

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 4 Jul 2012 03:18:19 -0700

Hi,

> will kill the performance.

Hmm. I have a naive question..
Let's say we send 100 packets and each packet has DSS option which
covers its own payload.
Now, if we lost every other packets before they arrived at the
receiver, I guess we'll need to maintain 50 mappings. So, I'm thinking
that even if we prohibit preemptive mappings, there might be a
situation we have to keep rather complex mappings..  Is this OK?

Thanks,
--
Yoshifumi

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Wed, 4 Jul 2012 13:39:52 +0300 (EEST)

> situation we have to keep rather complex mappings..  Is this OK?

It's not about storing those mappings until needed but having to do that 
independently of the actual payload. ...It depends on implementation 
detail of course, but I can easily see that the mapping could reside where 
the actual payload is until the payload itself is consumed and the 
mapping, if not pointing to some random place, will be needed at that 
point too and can be immediately discarded once used. But then if there's 
no payload to associate it with (either consumed already or not even 
arrived yet) you need start building other structures and do all kinds of 
extra "simple" operations such as allocate that space, maintain lists, do 
the memory accounting, and whatever...

-- 
 i.

---

From: Krishna Khanal <[email-masked]>
Date: Wed, 4 Jul 2012 16:53:10 +0530

Yoshifumi,
	I think this should not be a problem as long as you are handling out-of-order packets at subflow level. When every other packets are lost, this creates the out-of-order at the subflow level and you should not pass those packets to mptcp connection. This means you will always have a single map for a subflow or if the map is only for the payload carried by that packet, you don't have to store anything.

Regards,
Krishna

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Yoshifumi Nishida
Sent: Wednesday, July 04, 2012 3:48 PM
To: Christoph Paasch
Cc: [email-masked]; Mahesh M; Ilpo Järvinen
Subject: Re: [multipathtcp] Number of DSS to store

Hi,

Thanks,

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 04 Jul 2012 13:57:34 +0200

Hello,

just to confirm what Ilpo & Krishna said:

Our implementation does store segments in the subflow-out-of-order queue if 
there has been loss on the subflow.

And then, when the segment becomes in-order at the subflow-level and we can 
push it one level higher to the meta-socket we read the DSS out of the 
segment's TCP-option space. (for those who like to read code: 
http://goo.gl/WM7I7 - this is the place where we make the layer-switch from 
subflow to meta-socket and read the DSS-option out of the segment)

Cheers,
Christoph

On Wednesday 04 July 2012 03:18:19 Yoshifumi Nishida wrote:
> > The mapping is only needed after all the data has been received belonging
> > to this mapping (we have to wait for all the data, to verify the
> > DSS-csum).
> > 
> >> efficiently.> 
> > should then drop some of the received mappings. But if you drop them,
> > when the data comes in you don't know the mapping and have to drop the
> > data.

---

From: Anumita Biswas <[email-masked]>
Date: Wed, 04 Jul 2012 07:06:14 -0700

In Yoshifumi's example, if all the dropped packets belonged to one subflow, and all the other packets belonged to another sub flow, the MPTCP receiver would receive out of order data at the MPTCP level and would either have to retain the tcp options in each packet even after the packet reaches the MPTCP level or store the mappings in an auxiliary data structure.

If such striping isn't used across multiple subflows, then very little state needs to be stored at the receiver. But a robust receiver implementation should be prepared to handle out of order data and mappings at the MPTCP level.

Anumita.

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 04 Jul 2012 16:23:49 +0200

On Wednesday 04 July 2012 07:06:14 Anumita Biswas wrote:
> In Yoshifumi's example, if all the dropped packets belonged to one subflow,
> and all the other packets belonged to another sub flow, the MPTCP receiver
> would receive out of order data at the MPTCP level and would either have to
> retain the tcp options in each packet even after the packet reaches the
> MPTCP level or store the mappings in an auxiliary data structure.

Right. That's why we have two levels of out-of-order queues. One at the 
subflow-level and one at the MPTCP-level to support your described reordering.

The mapping is read when going from the subflow-level to the MPTCP-level and 
stored inside existing fields of the sk_buff. These fields are the seqno's of 
the subflow.
We overwrite these TCP subflow sequence-numbers as they are no more needed as 
they belonged to the subflow and we are moving a level higher to the MPTCP-
layer where only the data-sequence-number matters.

That way we avoid to increase the size of the sk_buff structure, as we reuse 
the TCP-fields for MPTCP-information. 

Cheers,
Christoph

> If such striping isn't used across multiple subflows, then very little state
> needs to be stored at the receiver. But a robust receiver implementation
> should be prepared to handle out of order data and mappings at the MPTCP
> level.

---

From: "Hampel, K Georg (K Georg)" <[email-masked]>
Date: Thu, 5 Jul 2012 08:47:03 -0500

It is not known how packet-re-segmenting middleboxes align DSS and payload. When packet-re-segmentation occurs the DSS mapping may arrive earlier at the receiver than the corresponding payload.

Therefore, as long as we subscribe to the assumption that packet-re-segmenting middleboxes exist, the implementation needs to support preemptive mappings.

Georg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Paasch
Sent: Wednesday, July 04, 2012 10:24 AM
To: Anumita Biswas
Cc: [email-masked]; Mahesh M; Ilpo Järvinen
Subject: Re: [multipathtcp] Number of DSS to store

Cheers,
Christoph

---

From: Christoph Paasch <[email-masked]>
Date: Thu, 05 Jul 2012 15:54:12 +0200

On Thursday 05 July 2012 08:47:03 Hampel, K Georg wrote:
> It is not known how packet-re-segmenting middleboxes align DSS and payload.
> When packet-re-segmentation occurs the DSS mapping may arrive earlier at
> the receiver than the corresponding payload.

How could this happen? Can you give an example with sequence-numbers, where 
the DSS is not on one of the packets whose sequence-numbers he belongs to.

What I mean is:

If a packet with DSS-mapping x -> y and subflow-seq-no x comes at the 
middlebox.
How could the middlebox resegment this packet so that the DSS-mapping is on 
packet with subflow-seq-no x-1 ?
This is a very buggy middlebox in my opinion, as it is sending a segment (x-1) 
that has already passed by this middlebox in the past.

Christoph

> Therefore, as long as we subscribe to the assumption that
> packet-re-segmenting middleboxes exist, the implementation needs to support
> preemptive mappings.

---

From: "Alan Ford (alanford)" <[email-masked]>
Date: Mon, 16 Jul 2012 16:21:21 +0000

All,

I've been considering how to reconcile the recent discussions in the draft
as it stands.

On Page 25 we already mention something in this area, but it does not
cover all cases we have discussed. In particular, it currently says:

  Implementations MAY hold onto such unmapped data for a short while in
  the expectation that a mapping will arrive shortly. [...]
  If a mapping for that subflow-level sequence space does
  not arrive within a receive window of data, that subflow SHOULD be
  treated as broken, closed with an RST, and any unmapped data silently
  discarded.

However, I'm not sure this really works. If a mapping was sent on a
duplicate ACK, this may not be discovered in time (by the lack of a DATA
ACK within said window). Maybe this actually needs to be one
2*receive_window?

Given the recent feedback, it looks like we also need to add more
constraints. For a simple requirement, how about we add:

  A mapping for subflow seqno x MUST not be sent before the mappings for
  subflow seqno 0 .. x-1 have been sent.

This shouldn't affect normal operation, since in the event of lost packets
the subflow will handle the re-ordering before it gets to MPTCP processing.

Later requirements discussed included:

  a) A mapping MUST appear, at latest, in the segment where is the first
  byte it covers is.

  b) If a DSS-mapping goes from x -> y in the subflow-seqno space, the
  DSS-option MUST be on one of these segments.

However, the latter of these prevents a duplicate ACK giving a mapping
being used before the data; and the former prevents preemptive mappings
entirely.

I do not see we have reached a particular consensus of what restrictions
we want to place on preemptive mappings. Can we refine anything here to
ensure a manageable and deterministic behaviour?

Regards,
Alan

---

From: "Alan Ford (alanford)" <[email-masked]>
Date: Wed, 1 Aug 2012 00:37:27 +0000

Hi all,

As just mentioned in the MPTCP WG, this is one issue that it would be
really good to come up with clarifications for the draft.

So, apart from the in-order delivery of mappings, are there any other
changes that we should make? What does implementation experience tell us
here?

Cheers,
Alan

---

From: Anumita Biswas <[email-masked]>
Date: Tue, 31 Jul 2012 21:44:35 -0700

A DSS option may not be sent with corresponding data if there isn't  enough tcp option space available such as due to the presence of SACK blocks.