Subject: Re: [multipathtcp] I-D Action: draft-ietf-mptcp-multiaddressed-06.txt

From: <[email-masked]>
Date: Fri, 3 Feb 2012 13:04:47 +0000

Hi
Thanks for your work to include the new Fast close - looks good.
We believe that this concludes all the outstanding actions, so we'll start a WG last call on this & on the API doc

Best wishes
Phil & Yoshifumi

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: 31 January 2012 17:58
To: [email-masked]
Cc: [email-masked]
Subject: [multipathtcp] I-D Action: draft-ietf-mptcp-multiaddressed-06.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Multipath TCP Working Group of the IETF.

	Title           : TCP Extensions for Multipath Operation with Multiple Addresses
	Author(s)       : Alan Ford
                          Costin Raiciu
                          Mark Handley
                          Olivier Bonaventure
	Filename        : draft-ietf-mptcp-multiaddressed-06.txt
	Pages           : 60
	Date            : 2012-01-31

   TCP/IP communication is currently restricted to a single path per
   connection, yet multiple paths often exist between peers.  The
   simultaneous use of these multiple paths for a TCP/IP session would
   improve resource usage within the network, and thus improve user
   experience through higher throughput and improved resilience to
   network failure.

   Multipath TCP provides the ability to simultaneously use multiple
   paths between peers.  This document presents a set of extensions to
   traditional TCP to support multipath operation.  The protocol offers
   the same type of service to applications as TCP (i.e. reliable
   bytestream), and provides the components necessary to establish and
   use multiple TCP flows across potentially disjoint paths.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-mptcp-multiaddressed-06.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-mptcp-multiaddressed-06.txt

_______________________________________________
multipathtcp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Christoph Paasch <[email-masked]>
Date: Mon, 20 Feb 2012 15:17:50 +0100

Hello,

sorry for writing this so late (8 days left until the end of the LC),
but I only now had the time to do a full read of the draft.

I have several comments concerning the draft as you can see below.

============
* Retransmission of the third ACK, to ensure reliable delivery of the
keys in MP_CAPABLE (page 14):

The retransmission of the third ack was introduced to allow lost third
acks, if the server is operating stateless. However, this introduced an
additional RTT before the client is able to send data and thus, we now
allow the client to send data before he receives the acknowledgment of
his third ack (in PRE_ESTABLISHED state).

I now realize, that the retransmission does not buy us a lot:

E.g., if the third ACK gets lost, but the client already starts to send
data and this data reaches the stateless server.
The server then can only send a subflow-ack without DATA_ACK and thus
behave as in regular TCP.
The client then should interpret this subflow-ack not as the one
acknowledging the third ack (the one with the MP_CAPABLE), but rather
fallback to regular TCP. Because, otherwise the client and server are no
more in-sync.

Thus, the retransmission only buys us something if the server is
behaving stateless, the third ack got lost and the server is the one
trying to send data first. This same problem is already present for
stateless servers operating with regular TCP.

I'm not sure that this current behavior of sending data in
PRE_ESTABLISHED state is safe (e.g., how to behave if the third ack and
the first data-packet got lost, but the second data-packet reaches the
server? - the server sends a duplicate subflow-ack without
MPTCP-options. How will the client interpret this subflow-ack?).

I think we should remove the retransmission of the third ack, as it does
not buy us a lot, and keep with the simpler and more straight-forward
mechanism of not caring about lost third-acks (as in draft v02).

============
* Note about pre-established state for *new* subflows (page 19, second
paragraph):

It is not clear to me, if we are allowed to send data on new subflows
while we are still in PRE_ESTABLISHED state. Because for the initial
subflow it is allowed.

If we are allowed to send data on new subflows while in PRE_ESTABLISHED,
it does not really make sense to differentiate between PRE_ESTABLISEHED
and ESTABLISHED for the additional subflows.

However, on page 21, we say that "If Host B does not receive the
expected MAC, or the MP_JOIN option is missing from the ACK, it MUST
close the subflow with a TCP RST."
Thus, I would say that we are not allowed to send data while in
PRE_ESTABLISHED state for new subflows. So, we should clearly specify
that this behavior is different for MP_JOIN's than for MP_CAPABLE's.

============
* Duplicate ACK's with an MPTCP-option should not be considered as
signals of congestion (first paragraph of page 13 and page 45,
"Duplicate ACK"):

I think, we should limit this to duplicate ACK's, containing either
REMOVE_ADDR, ADD_ADDR, MP_PRIO or MP_FAIL.

Otherwise, we have to make sure that we do not put DATA_ACK in duplicate
ack's (which is not so easy to do from an implementation point-of-view).
Additionally, it should be allowed to put a DATA_ACK in a duplicate ack,
because this DATA_ACK may advance the snd_una (due to data arrived on
another subflow).
And, the faster we get the info back for the new snd_una, the better it
is for MPTCP's performance (e.g., the other subflow may have a high RTT).

============
* page 25: "If a mapping for that subflow-level sequence space does not
arrive within a receive window of data, that subflow SHOULD be treated
as broken,..."

This is needed, because we allow to receive a DSS-mapping for a certain
subflow-sequence-space on a packet whose sub-sequence-numbers do not
belong to this space.

I suggest, that we do not support this due to several reasons:

1. why should an implementation decide to send the DSS-mapping on a
packet belonging to a different seq-space than specified in the
DSS-mapping-option.

2. if we do it, and we store a full receive-window, we have to use quite
a lot of memory before treating the subflow as broken.
If we would only support DSS-mappings on packets who belong to this
sequence-space, we can drop the data as soon as we receive a subsequence
packet on this subflow with a DSS-mapping that does not cover the
previous sequence-space (e.g., this happens in case of coalescing
middleboxes). And, if we don't receive mappings, we can drop data as
soon as we exceed 64KB, as the DATA_LEN's maximum is at 64KB.

============
* p.27, Section 3.3.3: "Note that when the DATA FIN is not attached to a
TCP segment containing data, the Data Sequence Mapping MUST have Subflow
Sequence Number of 0."

I don't see, why this is needed (we don't implement this - and
DATA_FIN's are still working very fine).
Couldn't we drop this paragraph for simplicity?

============
* p.29, Section 3.3.5: "It should only update its local receive window
values when the largest sequence number allowed increases."

Why not as in RFC 793, p.72:
"If SND.UNA < SEG.ACK =< SND.NXT, the send window should be updated.  If
(SND.WL1 < SEG.SEQ or (SND.WL1 = SEG.SEQ and SND.WL2 =< SEG.ACK)), set
SND.WND <- SEG.WND, set SND.WL1 <- SEG.SEQ, and set SND.WL2 <- SEG.ACK."

This is in my opinion more correct than the sentence in our draft. Maybe
we could just reference RFC793 in the mptcp-draft.

============
* p. 41: "If it is known that all unacknowledged data in flight is
contiguous..."

Does it really need to be contiguous?

Because, basically when falling back, the sender of the MP_FAIL
(data-receiver) will discard all data he receives after the
DATA-sequence-number which triggered the fallback. Only packets after
the infinite-mapping option will be allowed. Otherwise, we may have a
corrupted byte-stream presented to the application.
Thus, the data-sender (after receiving the MP_FAIL) will just send the
infinite-mapping option on the next packet. Irregardless, if the data
in-flight is contiguous or not.

============
Some minor nits:

p.8, Section 2:
"...between two hosts communicating hosts like..."

One "hosts" too much.

p.10, Section 2.4:
"...number space and a MPTCP option..." -> "...number space and an MPTCP
option..."

p.10, Section 2.4:
In the figure detailing the content of the DSS-option, DATA_LEN is missing.

p.26, Section 3.3.2:
"... to the behaviour of the standard TCP cumulative ACK in TCP SACK"
Not the ack in TCP SACK. It's just the usual standard TCP cumulative ACK.

p. 45:
"Instead an explicit data-level ACK is This separate subflow- and..."

This sentence got somehow mixed with something else.

Cheers,

Christoph

-- 
Christoph Paasch
PhD Student

IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://mptcp.info.ucl.ac.be
Université Catholique de Louvain
--

---

From: Alan Ford <[email-masked]>
Date: Tue, 21 Feb 2012 12:57:57 +0000

Hi Christoph,

Thank you for your feedback. Comments inline...

> I have several comments concerning the draft as you can see below.

It is true that if we removed the retransmission of the third ACK was added
for this corner case. But this is a well-known flaw in stateless operation
and SYN cookies, and our motivation for doing this was to make MPTCP's
handling of this better than regular TCP today.

I agree, however, that it would make things simpler, and faster in the
majority of circumstances, to revert to a simple three-way handshake.

Does anybody else have any thoughts?

Yes, the intention is that there is no data sent in the PRE_ESTABLISHED
state on additional subflows. This will be clarified.

Fair enough, that can be clarified easily enough, although I would rather
say "all MPTCP options except DSS" rather than explicitly list those signals
that it applies to.

One reason for this clause is to cope with the case if a sender needed to
send an MPTCP option urgently, but did not want to interrupt the data
stream. The sender would then send data before sending its mapping, which
would be covered by a mapping on a following packet.

I think it is needed.

This is needed: we must maintain the semantics of the data sequence space.
The DATA FIN does not map to subflow level sequence space, so we must set
that to zero.

Will need to think about that.

This is only talking about the case with a single subflow in use. In that
case, we want to be able to fall back without losing any data. This is to
cope with payload-changing middleboxes. In this case, the receiver does NOT
discard any data, and just carries on reading as if it was a standard TCP
session. But in order to do that, all data must be contiguous. This is a
safety clause more than anything else.

Thanks, will fix these.

Cheers,
Alan

>> A New Internet-Draft is available from the on-line Internet-Drafts
>> directories. This draft is a work item of the Multipath TCP Working Group of
>> the IETF.
>>

---

From: Christoph Paasch <[email-masked]>
Date: Wed, 22 Feb 2012 16:01:31 +0100

Hi Alan,

thanks for your reply.

Some comments inline:

> Does anybody else have any thoughts?

I realize one more thing, "conflicting" with the fact that no new
subflows may be established while in PRE_ESTABLISHED, as described in
section 3.1. Because, in section 3.6, we say that no new subflows are
allowed as long as we did not receive a DATA_ACK. And, we will always
enter PRE_ESTABLISHED before (or at the same time) receiving the first
DATA_ACK.

Thus, section 3.1 does not need to specify this, or may
forward-reference to section 3.6.

> say "all MPTCP options except DSS" rather than explicitly list those signals

"all MPTCP options except DSS" is fine for me.

> would be covered by a mapping on a following packet.

When would the sender want to send an MPTCP option urgently? E.g., for
ADD_ADDR the draft says "... Therefore, it is expected that an MPTCP
implementation will send the ADD_ADDR option on separate ACKs..."

I don't think that sending a duplicate ACK, holding the ADD_ADDR will
"interrupt" the data-stream. And, are ADD_ADDR's (or REMOVE_ADDR's) so
urgent?

The DSS-mapping has to be transmitted reliably, because otherwise we
will have plenty of segments without mapping lingering around in the
subflow-level receive-queues, and "consuming" space in the receive-window.
Thus, we have to put the DSS-mapping on a data-packet. If the
DSS-mapping-option does not map with the packet's subflow seq-space, we
will be consuming the "slot" of this packet's DSS-mapping, which in turn
again needs to be sent on the next data-packet. So, all DSS-mappings
will be "switched" by one packet.

Additionally, an attacker who manages to inject traffic on one subflow
(he correctly guesses the 32-bit subflow-sequence-numbers - but is not
able to do so for the 64-bit data-sequence-numbers) can fill up the
whole receive-window with dummy-packets that do not contain
DSS-mappings. Thus, he will considerably slow down the connection.

> I think it is needed.

Shouldn't we see if there is a real need for the "urgent MPTCP option"
you mentioned earlier, and if it's worth going down the pain to support
this ?

I believe the protocol will be more straight-forward and potentially
less memory-hungry if we don't support it.

> The DATA FIN does not map to subflow level sequence space, so we must set

Ok, fair enough.

> safety clause more than anything else.

I understand now, that the note of "contiguous" data in flight relates
to the data-level, and not to the subflow-level (I interpreted it to
belong to the subflow-level).

However, there is one more issue in this part of the draft:
It says, "(if the data is not contiguous, the sender MUST send an RST)".
However, sending a RST and establishing a new
(regular) TCP session, continuing the flow from the corrupted data-seq
number on, does not help. The receiver has to discard all data in the
out-of-order queue to avoid the following problem:

Suppose, we have two subflows, and the payload-modifying middlebox will
add one byte to the segments with data-seq-number 1-10. 1-10 has been
sent on subflow 1, and segment 11-20 has been sent on subflow 2.
Segment 11-20 are acked at the subflow-level and will be stored in the
data-level out-of-order queue because we don't yet have 1-10.

Due to the DSS-csum error on subflow 1, this one will get closed and
segment 1-10 will be reinjected on subflow 2.

Now, we only have one single subflow (2), and 1-10 will be sent on this
one, and the middlebox will again add one byte to this segment,
resulting in a DSS-csum failure and fallback to regular TCP.

1-10 will be sent again on subflow 2 and the middlebox adds the
byte, thus this segment becomes 1-11.

Now, we have a 1-byte overlap in the receive-queue with packets 1-11 and
(previously in the out-of-order queue) segment 11-20 and the byte-stream
has thus been corrupted.

The sender will retransmit segment 11-20 (at the reception 12-21 due to
the middlebox), but when this segment reaches the receiver, the
application may already have read the previous segments.

Thus, this is the reason why the receiver has to discard all
out-of-order data when announcing the fallback with MP_FAIL.
Additionally, it is not necessary that the sender sends a RST if the
data is not contiguous. It works by doing the direct fallback with the
infinite-mapping option.

Cheers,
Christoph

-- 
Christoph Paasch
PhD Student

---

From: Alan Ford <[email-masked]>
Date: Thu, 23 Feb 2012 12:30:00 +0000

Hi Christoph, all,

Responses inline...

> forward-reference to section 3.6.

Thanks, we'll change that - that simplifies things.

> "interrupt" the data-stream. And, are ADD_ADDR's (or REMOVE_ADDR's) so

I accept the scenarios where this could be deliberately used are low. There
is one scenario where this is currently expressly forbidden but could be
useful: for the very first packet of a connection. On the third ACK of the
initial subflow, we could send data and it will be covered by a DSS mapping
on a following packet. (We could make this a special case, however).

I cannot remember if we thought of any other reasons for this clause...

> data is not contiguous. It works by doing the direct fallback with the

Yes, you have a valid case there (if the same payload-altering middlebox is
operating on both subflows).

I think our intention has always been for all data to be discarded resent
from the data-level sequence number specified in MP_FAIL; however, this may
not be explicitly clear in the text. We shall improve this.

Regards,
Alan

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 24 Feb 2012 15:36:52 +0100

Hello Alan, all,

> Thanks, we'll change that - that simplifies things.

Good.

So, retransmission of the third ack will stay?

I just wanted to favor simplicity to make the protocol less complex. The
more complex MPTCP is, the harder it will be to get it deployed.

If the argument, that we want to fix with MPTCP the "broken" TCP
SYN-cookies with regular TCP, is strong enough to justify the additional
complexity, I'm fine with it.

> on a following packet. (We could make this a special case, however).

Regular TCP send's data on the third ACK, due to two reasons (unless I
missed one):
1. spare one packet
2. make the data reach slightly faster the application

However, for MPTCP these two arguments do not hold:
1. The DSS-mapping still needs to be sent on a separate packet. Thus, we
do not spare one packet.
2. We need the DSS-mapping before being able to push the data to the
application. (DSS-csum verification and we need the data-seq-number to
verify that the data is in-window). Thus, we still have to wait for the
DSS-mapping before pushing to the application.

Thus, what would be the incentive to put data on the third ack in case
of MPTCP ?

A scenario concerning missing DSS-mapping on packets:

If we are passing by a segment-coalescing middlebox, one mapping will be
lost of either of the two coalesced segments. The receiver has to store
this segment with the missing mapping due to our current design.

The sender is not aware of it. The subflow has acknowledged these
segments (at the subflow-level), and will continue with normal operation
until the meta-level retransmission timer will trigger the
retransmission of the data-segment. This retransmission will map to
another sequence-number space and will thus not resolve the mapping of
the original segment.

As the middlebox continues coalescing segments, the receiver will have
more and more of these in it's subflow-level receive-queues, waiting for
a covering DSS-mapping until the whole receive-buffer has been used up
by these "lingering" packets and they will finally get dropped. By the
time, as the receive-buffer filled up, the receive-window dropped close
to zero, and the sender is continuously receive-window blocked. Thus,
performance has dropped.

If we do not support the missing DSS-mapping, the receiver can drop the
"empty" segment as soon as the subflow-receive-queue receives a new
packet holding a DSS-mapping that does not cover the "empty" segment in
the receive-queue.

> not be explicitly clear in the text. We shall improve this.

Yes, making the discard explicit in the text will make it more clear.
Thanks!

Cheers,
Christoph

-- 
Christoph Paasch
PhD Student