Subject: [multipathtcp] MPCC and different transports

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 14 May 2009 21:04:09 +0200

In the BoF proposal it talks about "a set of architectural guidelines  
for congestion dependent multipath transport protocols". We now have  
two drafts for multipath TCP and SCTP could also be one of those with  
relatively small changes. And then DCCP with some more changes. And  
over beverages in Stockholm I can explain why I think cngestion  
dependent multipath is useful for inelastic traffic such as a/v  
streaming over RTP.

So if we have a shared architecture, does it make sense to implement  
the multipath congestion control up to five separate times?

Wouldn't it make more sense to define an abstraction layer between the  
transport protocol and the congestion control algorithm? That would  
make it much easier to make transports multipath-capable by simply  
linking them against the appropriate MPCC library. It would also make  
it much easier to work on multipath congestion control because the  
congestion control mechanism can then be a simple state machine that  
reacts to different events inside the transport protocol that the  
transport protocol exposes through a standardized interface rather  
than code spread all over the transport protocol implementation.

---

From: João Taveira Araújo <[email-masked]>
Date: Thu, 14 May 2009 23:57:20 +0100

Iljitsch van Beijnum wrote:
> transport protocol and the congestion control algorithm? 
 
This makes sense and reminds me of Bryan's case for decoupling CC and 
transport semantics in "Breaking up the transport logjam". But are we 
drifting away from the original motivation for extending TCP for 
multipath, namely for deployment purposes. Is the purpose of the 
eventual WG to work on multipath as a general concept and attempt to 
retrofit these results to transport protocols, or to take into account 
the limitations TCP and retain backwards compatibility whilst enabling 
multipath?

Cheers,
Joao.

---

From: Bryan Ford <[email-masked]>
Date: Fri, 15 May 2009 08:22:38 +0200

On May 15, 2009, at 12:57 AM, João Taveira Araújo <[email-masked]>  
wrote:

>> So if we have a shared architecture, does it make sense to  
>> implement the multipath congestion control up to five separate times?
>>
>> Wouldn't it make more sense to define an abstraction layer between  
>> the transport protocol and the congestion control algorithm?
> This makes sense and reminds me of Bryan's case for decoupling CC  
> and transport semantics in "Breaking up the transport logjam". But  
> are we drifting away from the original motivation for extending TCP  
> for multipath, namely for deployment purposes. Is the purpose of the  
> retrofit these results to transport protocols, or to take into  
> account the limitations TCP and retain backwards compatibility  
> whilst enabling multipath?

If the latter is the case, then I think that the scope of this WG  
should include ONLY single-ended TCP, because that's the only approach  
we know of so far that preserves any kind of backward compatibility.   
A two-ended MPTCP will encounter all the same deployment challenges  
that, say, CMT SCTP does, including the huge practical problem of  
legacy NATs in the path.

Bryan

>
>
> Cheers,
> Joao.
>
>
>
> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 15 May 2009 11:07:51 +0300

> that, say, CMT SCTP does, including the huge practical problem of

I believe two-ended version can be made to work over many NAT  
configurations, if you don't do the dynamic address management by  
explicitly passing the addresses in TCP options. I think this  
implicit address management was one of the variants discussed in the  
two-ended draft. This is important, because the ability to go through  
NATs is one of the most important benefits of doing this in TCP vs.  
other protocols in my opinion.

- Pasi

---

From: John Leslie <[email-masked]>
Date: Fri, 15 May 2009 08:11:25 -0400

Pasi Sarolahti <[email-masked]> wrote:
>> On May 15, 2009, at 12:57 AM, Jo??o Taveira Ara??jo <[email-masked]> wrote:
>>
>>> Is the purpose of the eventual WG to work on multipath as a general
>>> concept and attempt to retrofit these results to transport protocols,
>>> or to take into account the limitations TCP and retain backwards
>>> compatibility whilst enabling multipath?

   The latter seems more appropriate. (The former is too open-ended for
an IETF WG, though quite appropriate for an IRTF Research Group.)

>> we know of so far that preserves any kind of backward compatibility.

   I can't agree that we don't "know of" two-ended multipath approaches
that can preserve backward compatibility.

>> that, say, CMT SCTP does, including the huge practical problem of

   It's exactly things like NATs in the path that make the deployment
challenges so different!

> explicitly passing the addresses in TCP options.

   That is one promising approach. We will no doubt consider others.
Legacy NATs will almost certainly ignore stuff we might pass in TCP
options.

> I think this implicit address management was one of the variants
> discussed in the two-ended draft. This is important, because the
> ability to go through NATs is one of the most important benefits of
> doing this in TCP vs. other protocols in my opinion.

   The one-ended version, happily, goes through NATs at the unmodified
end quite nicely. Alton Ford's draft-ford-mptcp-multiaddressed proposes
an "Implicit Path Management" mode where "subflows" look to NATs just
like another TCP connection, but are "joined" to an existing flow and
the group of subflows appears as a single TCP connection to the
application layer.

   Thus Pasi and I have a (IMHO) reasonable belief that a mptcp WG
could design an extension to TCP that manages both one-ended and
two-ended multipath TCP with full NAT traversal.

   But if others think that belief unreasonable, perhaps we _should_
limit ourselves to the one-ended case...

   IMHO, discussing the architectural document in an open-ended way
is the right start. Committing to follow through on multipath for
TCP (only) feels safe enough to me. Committing to multipath for
other transports doesn't feel safe.

   YMMV, of course...

--
John Leslie <[email-masked]>

---

From: Bryan Ford <[email-masked]>
Date: Fri, 15 May 2009 14:11:31 +0200

On May 15, 2009, at 10:07 AM, Pasi Sarolahti wrote:
> two-ended draft. This is important, because the ability to go  
> through NATs is one of the most important benefits of doing this in  
> TCP vs. other protocols in my opinion.

How do you deal with TCP sequence numbers then?  Or by "many NAT  
configurations" did you mean only those that don't track TCP sequence  
numbers and expect them to work as they normally do in TCP?  How many  
NAT configurations is that, I wonder, especially given the increasing  
ubiquity of DPI technology that expects to see full TCP streams?  (For  
that matter, you don't even need NAT to have this problem: I bet many  
non-NAT firewalls will do something bad with TCP streams that appear  
to the firewall to be missing sequence numbers because they got sent  
on another path.)

Bryan

---

From: Costin Raiciu <[email-masked]>
Date: Fri, 15 May 2009 13:41:09 +0100

I also believe we can make the two-ended version work with NATs. If  
the one ended version could work, so will the two ended one. Details  
below.

There are three issues:
1. Will new option types be allowed through nats and other middleboxes?

If one new option type goes through, probably all will go through. I  
would personally bet new options do go through, but don't have  
numbers to back this up.
The options two ended multipath really needs are "mpath capable",  
"token", and "global sequence number". If it gets them through, it is  
superior to single-ended (see below at point 3).

If new options almost never make it through, two ended multipath must  
pretend it is single ended, even if both ends do this: it can't  
declare it supports mpath, there is no token and no global
sequence number; thus the two ended version is forced to stripe the  
global sequence space across the subflows it has, behaving like the  
single ended one.

2. address management

Here one can simply open subflows and see if they succeed (this is  
implicit management). If they don't it's maybe because the endpoint  
is behind a nat, so if we want to use the extra address it must be  
signaled otherwise to the other end. To get this working now, we must  
use options. For ipv4 addresses, there is still room in the options  
space to add at least 1 per packet. For ipv6, we can just use shim6  
to discover alternate paths; since we are talking about deploying  
now, let's leave ipv6 out of this.

Thus, if we can use new option types, we can do address management  
too for two ended multipath.

3. sequence numbers
The two-ended multipath has subflows which look over the wire as  
normal tcp connections, i.e. with contiguous sequence spaces, so they  
will pass through boxes like normalizers. The single ended one or CMT  
wont. Its unclear how many normalizers are deployed; Mark had  
understood from discussions with guys from the industry that many  
people use them (how many, don't know).

Finally, the one-ended one, and cmt for that matter, have the  
following issue: when a packet is lost and retransmitted on a  
different path, there is no way to know which of the paths delivered  
the packet unless one adds "options" to the packets. This is a more  
serious problem than not knowing whether the original or  
retransmitted packet got there in traditional tcp, as it places the  
host in the impossibility of deciding whether the path that carried  
the retransmitted packet has lost or delivered the packet.

Cheers,
Costin

---

From: "Ford, Alan" <[email-masked]>
Date: Fri, 15 May 2009 14:00:21 +0100

Bryan,

> to the firewall to be missing sequence numbers because they got sent

In draft-ford-mptcp-multiaddressed we describe using two sequence
spaces, so that each individual TCP connection that makes up a multipath
connection appears to be a complete and independent TCP connection to a
legacy NAT. The "implicit path management" as proposed in that draft
allows new connections to be added without the need for source address
knowledge.

Very interested in hearing you thoughts about feasibility issues, of
course!

Regards,
Alan

---

From: Bryan Ford <[email-masked]>
Date: Sat, 16 May 2009 06:50:25 +0200

> allows new connections to be added without the need for source address

OK, I see where you're going.  If you care about getting through DPI  
firewalls/NATs semi-reliably, though, I strongly suspect you'll have  
to scrap the resync packet mechanism described in section 4.4.3, as it  
seems extremely likely that many firewalls/NATs will balk in some way  
if you try to leave what looks to them like a hole in the TCP sequence  
space (which is what the resync mechanism will do).  Instead I see no  
reliable alternative but to make each subflow individually faithful to  
TCP semantics, including retransmitting any data that was previously  
transmitted on that subflow and lost - i.e., retransmitting the data  
_on the same subflow_ even if the data has since been successfully  
transmitted and ACKed on a different subflow.  Thus, shifting  
previously transmitted data among subflows would imply some  
semantically unnecessary data retransmissions, but this shouldn't be a  
performance problem as long as it doesn't happen too often, and it  
presumably shouldn't need to happen often if the transmit scheduler is  
designed with a bit of care.

That still leaves all the issues with the new TCP options.  Besides  
the issues you mention (e.g., having to fall back on single-path TCP  
if a NAT or firewall removes or rejects unknown options), there's a  
potentially more insidious one: what if a NAT or firewall allows  
unknown TCP options through but also mucks with the TCP sequence  
number space, e.g., by fragmenting, defragmenting, or realigning TCP  
segments along the way?  I don't have any concrete data on this, but  
since many middleboxes fully reassemble the original TCP byte stream  
in order to do reliable DPI, it seems very likely that they may treat  
the TCP stream's original segment boundaries as advisory at best.  For  
example, what if a middlebox receives a 2000-byte segment on its input  
side and fragments it into two 1000-byte segments for transmission on  
the smaller-MTU output side link, in the process simply duplicating  
any unkown TCP options?  The second output segment will then have an  
incorrect Data Sequence Number option (the same as the first, rather  
than the first + 1000 as it should be), confusing or possibly  
corrupting the MPTCP connection.  Similarly, what if two 1000-byte  
segments arrive at the middlebox out of order on a low MTU link, the  
middlebox buffers the first until the hole is filled so it can do DPI  
properly, and then it retransmits that data as one 2000-byte segment  
on the higher-MTU outgoing link?  Those two 1000-byte segments might  
have had two different unrelated Data Sequence Numbers from MPTCP's  
perspective, but only one of those sequence numbers will get through,  
and the second 1000 bytes will get put in the wrong place in the  
higher-level stream.

It might arguably be reasonable for us to hope (and pray!!!) that all  
or practically all deployed middleboxes either (a) drop all unknown  
TCP options, which castrates MPTCP by falling back on single-path mode  
but at least still works, or (b) pass through unknown TCP options but  
never modify the sequence numbering.  It would be nice if this were  
the case, and might be a semi-reasonable expectation based on the  
likelihood that most middleboxes either process TCP segments packet-by- 
packet without doing much if anything to sequence numbers, or else  
fully split/proxy the TCP connection and in the process throw away  
unknown options since they're effectively acting as a TCP endpoint.   
But given all the innumerable types of weird behavior we know  
middleboxes to exhibit, this strikes me as wishful thinking.  At the  
very least, someone would need to do a fairly large-scale test of the  
behavior of TCP with unknown options over a wide variety of deployed  
NATs, before we could have much confidence that this approach to an  
MPTCP would be reliable.  Especially since the potential cost of  
guessing wrong on this count includes not just reversion to single- 
path behavior or even lost connectivity, but silent (and possibly very  
non-deterministic) application data corruption!

Bryan

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 16 May 2009 14:48:27 +0200

On 15 mei 2009, at 14:11, Bryan Ford wrote:

> configurations" did you mean only those that don't track TCP  
> sequence numbers and expect them to work as they normally do in  
> TCP?  How many NAT configurations is that, I wonder, especially  
> given the increasing ubiquity of DPI technology that expects to see  
> full TCP streams?

Yes, that's a good question.

On the one hand it could be that they insist on seeing all packets.  
However, even if the NAT/firewall saw the packet, the destination may  
not see it because it gets lost between the NAT/FW and the  
destination. So the window of sequence number space that is allowed  
through must track the acknowledgments from the destination. So that  
would suggest that there is a good chance that if the destination ACKs  
something that the NT/FW didn't see the window still progresses and  
there is no trouble.

In this case, it would be possible to use a single sequence number  
space if we make sure that the NAT/FWs get to see enough packets to  
progress their windows, which would be about one per path per window.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 16 May 2009 15:12:04 +0200

On 15 mei 2009, at 14:41, Costin Raiciu wrote:

> since we are talking about deploying now, let's leave ipv6 out of  
> this.

Absolutely not.

IPv4 is walking on its last legs, deploying new IPv4-only technology  
now is almost criminal.

> the retransmitted packet has lost or delivered the packet.

There are two types of retransmissions: fast and RTO-based.

In the case of a packet being delivered after the RTO expires and a  
retransmission has occurred, I think we can safely just pretent the  
original packet was lost, because it was delayed _a lot_.

In theory then the retransmission on another path could be lost and we  
progress the cwin of that path incorrectly, but the circumstances  
under which this would happen are very unusual: a very large RTT  
increase on one path and then being extremely unlucky (normal packet  
loss is < 1%) or having massive congestion on the other path. If it's  
congestion we can safely assume another packet will drop and the cwin  
will shrink posthaste.

The case where a packet that was presumed to be lost after fast  
retransmit was triggered and then delivered after all should be more  
common, as it doesn't require extremely large RTT increases.

However, this should still be rare because it requires some  
significant reordering in the network.

Because reordering is actively avoided, it makes sense that when it  
occurs, it's in large amounts because of a network path that uses  
parallelism that doesn't care about reordering so it will happen a  
lot, which would be bad for performance because of the constant cwin  
halvings.

However, in that case, we shouldn't just see regular occurrences of  
three duplicate ACKs, but also larger numbers of single and double  
duplicate ACKs. Statistically, the difference between reordering and  
losses should be fairly obvious.

Loss (assuming 6 pkts before a fast retransmitted packet is received):

* * * * * *
* * * * * *
* * * * * * *
* * * * * * *
1 2 3 4 5 6 7 8 9

Reordering:

*
*
*
*
* *
* *
* * *
* * * *
1 2 3 4 5 6 7 8 9

The reordering issue can be solved by simply upping the fast  
retransmit threshold. There are a few papers that look at this. Not  
sure if they make the behavior dynamic to preserve fast retransmit as  
much as possible, though.

If you use a multiaddress multipath solution and you retransmit over a  
different path, depending on how the ACKs work, you may be able to  
detect the difference between the original packet and the  
retransmission through the addresses.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 16 May 2009 15:22:11 +0200

On 16 mei 2009, at 6:50, Bryan Ford wrote:

> That still leaves all the issues with the new TCP options.

A discussion that a bunch of us had offline: would it be useful to  
exchange signaling information in-band at the beginning of a subflow?

I.e., I set up something that looks like a new TCP session to  
middleboxes, but in reality it's an additional subflow for an existing  
session. Rather than negotiate all the housekeeping in options, I  
simply do that in-band. When the negotiation is finished, I start  
exchanging data.

In fact, I could do that for all packets. The first n bytes following  
the TCP header could be the second sequence number.

Obviously deep packet inspection boxes wouldn't like that, but if we  
can't do options, we must maintain the sequence numbering and we can't  
touch the data then maybe using that path for multipath isn't such a  
good idea after all.

> middleboxes to exhibit, this strikes me as wishful thinking.

The good thing here is that we don't really care that we can't get  
through a given middlebox. Obviously we want to get through as many as  
we can in order to be able to use as many paths as we can, but we  
always have the default path so losing additional paths is something  
that we can live with. The only issue would be middleboxes that do  
stuff that makes MPTCP work incorrectly. This could happen when the  
options and sequence numbers get out of alignment like you describe,  
so this situation may need to be detected.

---

From: John Leslie <[email-masked]>
Date: Sat, 16 May 2009 09:44:22 -0400

Bryan Ford <[email-masked]> wrote:
>> spaces, so that each individual TCP connection that makes up a  
>> multipath connection appears to be a complete and independent TCP
>> connection to a legacy NAT. The "implicit path management" as
>> proposed in that draft allows new connections to be added without the
>> need for source address knowledge.
> 
> If you care about getting through DPI firewalls/NATs semi-reliably,
> though, I strongly suspect you'll have to scrap the resync packet
> mechanism described in section 4.4.3, as it seems extremely likely
> that many firewalls/NATs will balk in some way if you try to leave
> what looks to them like a hole in the TCP sequence space (which is
> what the resync mechanism will do)...

   We're getting far beyond the usual BoF stage here -- speculating
on what (future?) NATs might do. (I don't believe checking for holes
in TCP sequence space is common today -- how would you ever recover
synchronization if the NAT starts diddling?)

> I don't have any concrete data on this, but since many middleboxes
> fully reassemble the original TCP byte stream in order to do reliable
> DPI,

   It seems unlikely that all that "many" middleboxes do deep packet
inspection. And I most sincerely hope that we don't include waging
an arms-escalation war with deep-packet-inspection in our charter.
Deep-packet-inspection is something governments do, and they're
likely to take offense if we try to play in that playground. :^(

> it seems very likely that they may treat the TCP stream's original
> segment boundaries as advisory at best. For example, what if a
> middlebox receives a 2000-byte segment on its input side and fragments
> it into two 1000-byte segments for transmission on the smaller-MTU
> output side link, in the process simply duplicating any unkown TCP
> options?

   I can barely imagine how many things that would break!

> ... It might arguably be reasonable for us to hope (and pray!!!)
> that all or practically all deployed middleboxes either (a) drop
> all unknown TCP options, which castrates MPTCP by falling back on
> single-path mode but at least still works, or (b) pass through
> unknown TCP options but never modify the sequence numbering.

   NAT traversal is being considered in many IETF WGs. I haven't
heard this level of speculation in other WGs. You are describing
things that do serious violence to the very _basis_ of TCP; and
you seem to be suggesting that we should avoid any changes to TCP
that might be broken by broken middleboxes. I dissent.

> ... But given all the innumerable types of weird behavior we know  
> MPTCP would be reliable.

   ... which I believe is the intent of our charter: making an
Experimental protocol.

---

From: Bryan Ford <[email-masked]>
Date: Sat, 16 May 2009 21:23:05 +0200

On May 16, 2009, at 2:48 PM, Iljitsch van Beijnum wrote:
> However, even if the NAT/firewall saw the packet, the destination  
> may not see it because it gets lost between the NAT/FW and the  
> would suggest that there is a good chance that if the destination  
> ACKs something that the NT/FW didn't see the window still progresses  
> and there is no trouble.

I agree that this is fairly reasonable behavior for a middlebox that  
merely tracks TCP flows on a packet-by-packet basis and doesn't need  
to or try to reconstruct the original semantic byte stream that's  
being transmitted via TCP.  But some middleboxes do need to  
reconstruct the original byte stream: e.g., firewalls that parse HTTP  
or other application-level headers and filter traffic based on them;  
virus scanners that scan for virus signatures within payloads, etc.   
If such a firewall has a policy that involves analyzing the  
reconstructed TCP byte stream as a condition on allowing the  
connection to continue or complete, then it seems unlikely that the  
firewall would just wave on through any holes in the sequence number  
space: more likely, after filling its reorder buffer space it will  
simply block the stream until the holes are filled, which may be never  
if the stream is an MPTCP stream.

Further, many middleboxes even have the capability to insert or remove  
bytes in TCP streams, and supporting that capability entails not only  
reconstructing the original byte stream but delaying the forwarding of  
segments received out of order until the "hole" is filled.  For  
example, if the NAT were to receive segment 2000-2999 out of order,  
forward it, and then upon receiving a prior segment 1000-1999 only  
then discover that it needs to insert 15 bytes somewhere in that  
segment, it's screwed because segment 2000-2999 should have had its  
sequence numbers translated to 2015-3014.  Thus such a NAT _MUST_  
forward the segments in order in the off chance that it might have to  
translate the sequence numbers - even if on most streams its "trigger  
condition" for doing such a modification will never be activated.   
NATs have been doing this to FTP control connections for years, and  
we've all heard of the evil ISPs that recently started deploying boxes  
that insert ads into the HTML payloads of HTTP connections their  
customers make to random web sites.

I'm not in any way trying to defend such practices in general, and we  
might come up with reasons to hope that such TCP stream modification  
capabilities would never actually be triggered on an MPTCP stream  
masquerading as a TCP stream - but just the fact that many middleboxes  
have such capabilities means that they HAVE TO have been designed with  
the ability to buffer out-of-order packets and wait for holes to be  
filled before forwarding data, and it's anyone's guess under what  
(perhaps even unnecessary) conditions this capability in a particular  
NAT vendor's box might be in effect on random TCP connections, causing  
hole-filled MPTCP streams to grind suddenly to a halt.

I'm not claiming to know whether this is really a big, small, or  
nonexistent problem in practice: I'm just saying that it's a concern  
that seams realistic, we don't yet have the data to answer that  
question, and it seems like we would need to before we should commit  
to an MPTCP design that fills TCP connections with holes.

Bryan

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 16 May 2009 22:54:27 +0200

On 16 mei 2009, at 21:23, Bryan Ford wrote:

> But some middleboxes do need to reconstruct the original byte  
> stream: e.g., firewalls that parse HTTP or other application-level  
> headers and filter traffic based on them; virus scanners that scan  
> for virus signatures within payloads, etc.

Ugh!

However, at this stage we're beyond regular NAT and even typical  
firewall behavior.

The trouble is that we would have to detect this kind of behavior to  
disable the affected subflows because otherwise the integrity of the  
data stream would be compromised. (Why is it "compromised" but  
"analyzed"? English is so infuriatingly inconsistent.)

I guess a good way to do this would be with some kind of HMAC  
protection of the data. (There goes the option space...)

[...]

> NATs have been doing this to FTP control connections for years,

Right, and with stuff like SIP and RTSP. However, this is all low- 
volume control stuff, so if we can detect this we can disable  
multipath for it without ill effects.

> and we've all heard of the evil ISPs that recently started deploying  
> boxes that insert ads into the HTML payloads of HTTP connections  
> their customers make to random web sites.

That's nature's way of telling people that they should use encryption...

[...]

> causing hole-filled MPTCP streams to grind suddenly to a halt.

Perhaps this is a good opportunity for Costin to remind us why it was  
a bad idea to send a packet with the seqnum that would fill the hole,  
but now with different data? It seems that that would solve this  
particular issue.

---

From: Costin Raiciu <[email-masked]>
Date: Sat, 16 May 2009 23:02:36 +0100

I agree with Bryan that there are many failure modes, and one of the  
reasons to not retransmit different data with the same seq no was the  
fact that traffic normalizers cache seqno-data mappings and replay  
the original (cached) data  instead of the new data when the same  
seqno arrives - if the packet was lost after the normalizer, than  
this corrupts application data. Also, when you get an ack for that  
data you don't know whether the original data arrived (that maybe got  
lost) or the new one (remember, there is no data ack).  The resync  
packet is just an optimization to avoid resending useless packets,  
but and it could be scrapped it for compliance.

I admit had not thought about the fragmentation/reassembly scenario  
before. Iljitsch's proposal to do in band signalling doesn't help  
there either; when you fragment a packet whose data begins with a  
data seq num, the second packet will not have the proper seq num.

I can think of a few heuristics to detect when such fragmentation  
reassembly happens, but am not sure they cover all cases. To be able  
to deterministically tell whether the packet has been fragmented one  
fix would be to include the data size as yet another option. That  
seems like an overkill :(.

Cheers,
Costin

On 16 May 2009, at 21:54, Iljitsch van Beijnum wrote:

> The trouble is that we would have to detect this kind of behavior  
> to disable the affected subflows because otherwise the integrity of  
> the data stream would be compromised. (Why is it "compromised" but  
>> and we've all heard of the evil ISPs that recently started  
>> deploying boxes that insert ads into the HTML payloads of HTTP  
>> connections their customers make to random web sites.
>
> Perhaps this is a good opportunity for Costin to remind us why it  
> was a bad idea to send a packet with the seqnum that would fill the  
> hole, but now with different data? It seems that that would solve  
> this particular issue.

---

From: Michael Tüxen <[email-masked]>
Date: Sun, 17 May 2009 11:59:30 +0200

On May 15, 2009, at 8:22 AM, Bryan Ford wrote:

>> for multipath, namely for deployment purposes. Is the purpose of  
>> the eventual WG to work on multipath as a general concept and  
>> attempt to retrofit these results to transport protocols, or to  
>> take into account the limitations TCP and retain backwards  
> should include ONLY single-ended TCP, because that's the only  
> approach we know of so far that preserves any kind of backward  
> compatibility.  A two-ended MPTCP will encounter all the same  
> deployment challenges that, say, CMT SCTP does, including the huge  
> practical problem of legacy NATs in the path.
Hi Bryan,

just a note: The "problem" is not with CMT SCTP and NATs, but with  
multihoming an NATs.
Please note that this problem can be solved by using a NAT as  
described in
http://www.ietf.org/internet-drafts/draft-ietf-behave-sctpnat-01.txt
or one can use UDP encapsulation. Both mechanisms are implemented in  
FreeBSD.
(which also has an implementation of CMT).

---

From: Bryan Ford <[email-masked]>
Date: Sun, 17 May 2009 19:20:27 +0200

On May 16, 2009, at 3:22 PM, Iljitsch van Beijnum wrote:
> middleboxes, but in reality it's an additional subflow for an  
> existing session. Rather than negotiate all the housekeeping in  
> options, I simply do that in-band. When the negotiation is finished,  
> I start exchanging data.
>
> In fact, I could do that for all packets. The first n bytes  
> following the TCP header could be the second sequence number.
>
> can't do options, we must maintain the sequence numbering and we  
> can't touch the data then maybe using that path for multipath isn't  
> such a good idea after all.

This approach sounds more promising to me in general.  Then MPTCP  
starts to take more and more of the flavor of a new transport protocol  
layered atop an existing protocol (namely legacy single-path TCP),  
taking advantage of legacy TCP to provide per-flow congestion control  
and get through middleboxes.  From another angle, in the architectural  
view Jana and I proposed in our HotNets paper, it sounds like MPTCP is  
effectively using legacy TCP as a Flow Layer protocol, providing per- 
flow congestion control (and middlebox compatibility) but implementing  
a new inter-flow level of sequencing and retransmission to provide the  
new, "real" transport protocol.

If MPTCP builds atop legacy TCP mostly as-is, using TCP as a Flow  
Layer (perhaps with enhancements to legacy TCP's congestion control to  
address the fairness/aggressiveness/resource pooling issues), and if  
MPTCP doesn't play too fast and loose with legacy TCP's options or  
sequencing, then individual MPTCP flows should even be able to get  
through fancier middleboxes like Performance Enhancing Proxies (PEPs).

This approach would also have the nice benefit of effectively  
restoring the long-lost "end-to-endness" of TCP's originally intended  
semantics, from the perspective of the application, even when many or  
all of the individual legacy TCP flows atop which an MPTCP connection  
is built are being split in the middle by stateful firewalls, NATs,  
and/or PEPs.  For example, if a stateful middlebox crashes and loses  
its legacy TCP state, the legacy TCP connection is hosed, but the  
MPTCP connection might well be able to notice the TCP flow's  
connection reset and just retry establishing that flow, while shifting  
traffic to other flows in the meantime.

Taking this approach further, have you thought about how MPTCP should/ 
might interact with IPSEC?  Of course IPSEC can always be deployed  
above IP and below legacy TCP, as usual, but we know how well IPSEC  
tends to get along with middleboxes.  Combining traditional IPSEC with  
MPTCP would effectively defeat all the care MPTCP went to to provide  
backward compatibility by "looking like" legacy TCP.  Even with  
IPSEC's recent UDP encapsulation enhancements to get through NATs and  
firewalls, it remains incompatible with PEPs or TCP proxies that need  
to split or monitor TCP's congestion control loop, not to mention  
middleboxes that let through only TCP and not UDP.  But taking the  
view of legacy TCP as a Flow Layer when used by MPTCP, the right place  
to deploy IPSEC is actually _above_ legacy TCP but below MPTCP's inter- 
flow transport layer.  This way, "IPSEC-secured MPTCP" would have all  
the same backward compatibility advantages of unsecured MPTCP because  
the Flow Layer part looks just like ordinary TCP, but the IPSEC layer  
in the middle would then be able to enforce the true "end-to-endness"  
of the MPSEC transport layer by preventing middleboxes from  
interfering with or even seeing the MPSEC-level data transmission loop.

> through a given middlebox. Obviously we want to get through as many  
> as we can in order to be able to use as many paths as we can, but we  

I agree that it's not disaster if MPTCP doesn't manage to run  
multipath through certain types of middleboxes - although if backward  
compatibility is one of MPTCP's primary design goals, it's obviously  
best if it can run multipath through as wide a variety of middleboxes  
as possible, and I believe the above approach would achieve that.

Cheers,
Bryan

---

From: Bryan Ford <[email-masked]>
Date: Sun, 17 May 2009 19:33:50 +0200

> synchronization if the NAT starts diddling?)

We're not talking about future NATs here: almost all NATs already have  
the capability to reconstruct the byte stream and diddle with sequence  
numbers (e.g., for FTP control connections, which NATs have been doing  
for years).  This capability might currently only get activated under  
certain very specialized conditions that we might hope would never  
include MPTCP flows, but we just don't know until we start testing.   
And as I said earlier, the problem here is that the price for getting  
this decision wrong even only once in a very rare while can be  
extremely high: e.g., silent nondeterministic corruption of  
application data.

> likely to take offense if we try to play in that playground. :^(

Deep packet inspection is something practically all state-of-the-art  
corporate firewalls do.  Take a look at the marketing literature for  
McAfee Sidewinder just as one example.  In fact I suspect if you asked  
a firewall marketing person whether a non-DPI firewall could be  
marketed today, I expect they'd say no.

>   I can barely imagine how many things that would break!

Why would it break anything?  It would just be doing the same things  
the original TCP sender would do, while remaining perfectly consistent  
with TCP's byte stream semantics.  (Of course it breaks TCP's end-to- 
endness, but all stateful middleboxes - i.e., practically all  
middleboxes - already do that, and we don't like that but have learned  
to live with it.)

> that might be broken by broken middleboxes. I dissent.

If by "the very _basis_ of TCP" you refer to TCP's end-to-end  
principle, then middleboxes already do serious violence to that.  We  
can define DPI middleboxes as broken, but that's just denial of  
reality: they're here, they're widely deployed, and we have to live  
with them.

>   ... which I believe is the intent of our charter: making an

Granted.

Bryan

---

From: Janardhan Iyengar <[email-masked]>
Date: Sun, 17 May 2009 16:06:07 -0400

Bryan,

> firewalls/NATs semi-reliably, though, I strongly suspect you'll have to 
> scrap the resync packet mechanism described in section 4.4.3, as it 
> seems extremely likely that many firewalls/NATs will balk in some way if 
> you try to leave what looks to them like a hole in the TCP sequence 
> transmitted on that subflow and lost - i.e., retransmitting the data _on 
> the same subflow_ even if the data has since been successfully 
> transmitted and ACKed on a different subflow.  Thus, shifting previously 
> transmitted data among subflows would imply some semantically 
> unnecessary data retransmissions, but this shouldn't be a performance 
> problem as long as it doesn't happen too often, and it presumably 
> shouldn't need to happen often if the transmit scheduler is designed 
> with a bit of care.

There is another way to deal with holes in the sequence space---divorce the data that is being sent from the associated sub-flow sequence number. Data needs to be associated with only Data-Sequence-Numbers.  Thus what appears to be a retransmission at the flow level can in reality be a new data transmission, and the actual retransmission could have been transmitted over a different subflow with a new subflow sequence number.

This does need more changes to the TCP state machine, but it is an alternative that has been explored before in pTCP (http://www.ece.gatech.edu/research/GNAN/work/ptcp/ptcp.html)

- jana

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar

---

From: Janardhan Iyengar <[email-masked]>
Date: Sun, 17 May 2009 16:18:04 -0400

> 3. sequence numbers
> The two-ended multipath has subflows which look over the wire as normal 
> tcp connections, i.e. with contiguous sequence spaces, so they will pass 
> through boxes like normalizers. The single ended one or CMT wont. Its 
> unclear how many normalizers are deployed; Mark had understood from 
> discussions with guys from the industry that many people use them (how 
> many, don't know).

This is true: CMT will not appear contiguous in sequence space to middleboxes.  Are "normalizers" middleboxes that reconstruct flows?

> Finally, the one-ended one, and cmt for that matter, have the following 
> issue: when a packet is lost and retransmitted on a different path, 
> there is no way to know which of the paths delivered the packet unless 
> one adds "options" to the packets. This is a more serious problem than 
> not knowing whether the original or retransmitted packet got there in 
> traditional tcp, as it places the host in the impossibility of deciding 
> whether the path that carried the retransmitted packet has lost or 
> delivered the packet.

I don't understand why this retransmission ambiguity is any worse with CMT than with regular TCP.  TCP also tries to determine whether "the path that carried the retransmitted packet has lost or delivered the packet", TCP just happens to use only one path on which it tries to determine if a loss occurred. This is no different in CMT, and the same disambiguation methods that are employed for TCP can be used for CMT.

- jana

---

From: Costin Raiciu <[email-masked]>
Date: Sun, 17 May 2009 22:39:37 +0100

On 17 May 2009, at 21:18, Janardhan Iyengar wrote:

>> normal tcp connections, i.e. with contiguous sequence spaces, so  
>> they will pass through boxes like normalizers. The single ended  
>> one or CMT wont. Its unclear how many normalizers are deployed;  
>> Mark had understood from discussions with guys from the industry  
>> that many people use them (how many, don't know).
>
> This is true: CMT will not appear contiguous in sequence space to  
> middleboxes.  Are "normalizers" middleboxes that reconstruct flows?
Yes.
>> different path, there is no way to know which of the paths  
>> delivered the packet unless one adds "options" to the packets.  
>> This is a more serious problem than not knowing whether the  
>> original or retransmitted packet got there in traditional tcp, as  
>> it places the host in the impossibility of deciding whether the  
>> path that carried the retransmitted packet has lost or delivered  
>> the packet.
>
> I don't understand why this retransmission ambiguity is any worse  
> with CMT than with regular TCP.  TCP also tries to determine  
> whether "the path that carried the retransmitted packet has lost or  
> delivered the packet", TCP just happens to use only one path on  
> which it tries to determine if a loss occurred. This is no  
> different in CMT, and the same disambiguation methods that are  
> employed for TCP can be used for CMT.
It's not exactly the same thing. When you retransmit on the same  
path, when you get the ack you know this path delivered one the  
packets for sure, so its working. This is not the case in multipath:  
say you have a single packet, it gets lost on one path, you send it  
down the other, and you get an ack. Which path delivered it? It can  
be that the second path is dead, and the first path had a (really  
bad) hickup, so assuming the second path delivered the packet is not  
necessarily correct.

Determining which path works is important to schedule the following  
packets, otherwise the whole connection can just grind to a halt. Its  
important to deterministically know which path delivered the packet.  
In the single seq space design, if one doesn't add extra options (or  
the like), the only way to achieve this is restricting  
retransmissions to be on the same flow.

Cheers,
Costin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 18 May 2009 13:29:03 +0200

> starts to take more and more of the flavor of a new transport  
> protocol layered atop an existing protocol (namely legacy single- 
> path TCP), taking advantage of legacy TCP to provide per-flow  
> congestion control and get through middleboxes.

Right. The multiaddress multipath TCP as outlined in the two-ended  
draft already seems a bit like TCP-over-TCP.

> Even with IPSEC's recent UDP encapsulation enhancements to get  
> through NATs and firewalls, it remains incompatible with PEPs or TCP  
> proxies that need to split or monitor TCP's congestion control loop,

Isn't that a feature?

> not to mention middleboxes that let through only TCP and not UDP.   
> But taking the view of legacy TCP as a Flow Layer when used by  
> MPTCP, the right place to deploy IPSEC is actually _above_ legacy  
> TCP but below MPTCP's inter-flow transport layer.

Isnt't that basically what SSL/TLS does today?

The main advantage of IPsec over TLS is that IPsec also protects  
against attacks against TCP, IPsec-over-TCP wouldn't have that  
protection for the outer TCP session. However, the multipath layer  
would be able to recover from successful attacks so this protection  
wouldn't be as crucial as in the single path case (unless all paths  
are being attacked). So TLS may suffice.

An interesting approach would be TCP over TLS over TCP where the TLS  
protects against the issues with middleboxes rewriting data that we  
talked about. To do this well, it would be helpful to send each TLS  
frame/packet (not sure what the right terminology is) over the same  
path.

Iljitsch

---

From: "Ford, Alan" <[email-masked]>
Date: Mon, 18 May 2009 16:32:44 +0100

Bryan Ford wrote:
> and get through middleboxes.  

However, simply putting data at the start of a packet still relies on
middleboxes not to mess with the segmentation. It would be nice to get a
solution where it doesn't matter if this happens.

Which makes me wonder whether a TLS-like solution, where all data (both
application and control) is carried in the TCP stream, and data is
chunked and preceded by a header of type, length, and data sequence
number (if appropriate). We could then switch between control data and
application data with no major issues, and we still get the benefits of
data sequence numbering without TCP Options. I haven't thought this
through yet, so it may be riddled with holes, so fire away...!

> and/or PEPs.  

I'm not sure if that really adds anything over and above what we already
have. Although the MPTCP layer would be (at the moment - who knows what
NATs may do in the future!) end-to-end, its primary purpose is to
reassemble data spread across multiple paths, and there is little else
in the protocol at that level (of course, this could be a convenient
place for future extensions, but that's not our concern). Data as
delivered to the application would be pretty much the same as today.

> For example, if a stateful middlebox crashes and loses

Indeed, that's one particular use case for MPTCP, and should work in
either of the proposals put forward so far.

> interfering with or even seeing the MPSEC-level data transmission
loop.

An interesting thought, but would IPSec-over-MPTCP be any more
appropriate than using TLS? (Aside from being transparent to the
application, of course).

Assuming paths with no NATs and thus IPSec end-to-end, it should be
feasible to run MPTCP over TCP over IPSec with no issues. But I must
admit I haven't gone through this step-by-step to make sure!

Cheers,
Alan

---

From: Janardhan Iyengar <[email-masked]>
Date: Mon, 18 May 2009 15:21:34 -0400

> through yet, so it may be riddled with holes, so fire away...!

This does sound interesting... if you could define specific "control" chunks, then you could get rid of almost all of the TCP options, except for negotiation of MPTCP.  It is better than having bytes at the head of segments that can be lost during segmentation in the middle.

I'll note that the TCP stream now looks even more like an SCTP association (data and control chunks in SCTP are exactly chunks in the SCTP association with TLV headers and sequence numbers where appropriate.)

- jana

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 09:36:10 +0200

On May 17, 2009, at 10:06 PM, Janardhan Iyengar wrote:
> Bryan,
>
>> OK, I see where you're going.  If you care about getting through  
>> DPI firewalls/NATs semi-reliably, though, I strongly suspect you'll  
>> have to scrap the resync packet mechanism described in section  
>> 4.4.3, as it seems extremely likely that many firewalls/NATs will  
>> balk in some way if you try to leave what looks to them like a hole  
>> in the TCP sequence space (which is what the resync mechanism will  
>> do).  Instead I see no reliable alternative but to make each  
>> subflow individually faithful to TCP semantics, including  
>> retransmitting any data that was previously transmitted on that  
>> subflow and lost - i.e., retransmitting the data _on the same  
>> subflow_ even if the data has since been successfully transmitted  
>> and ACKed on a different subflow.  Thus, shifting previously  
>> unnecessary data retransmissions, but this shouldn't be a  
>> presumably shouldn't need to happen often if the transmit scheduler  
>> is designed with a bit of care.
>
> There is another way to deal with holes in the sequence space--- 
> divorce the data that is being sent from the associated sub-flow  
> sequence number. Data needs to be associated with only Data-Sequence- 
> Numbers.  Thus what appears to be a retransmission at the flow level  
> can in reality be a new data transmission, and the actual  
> retransmission could have been transmitted over a different subflow  
> with a new subflow sequence number.
>
> This does need more changes to the TCP state machine, but it is an  
> alternative that has been explored before in pTCP (http://www.ece.gatech.edu/research/GNAN/work/ptcp/ptcp.html 
> )

A clever trick, but it seems this approach would still be subject to  
corruption by over-intelligent middleboxes (I guess we're calling them  
"normalizers"?)...

Bryan

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 10:16:19 +0200

On May 18, 2009, at 1:29 PM, Iljitsch van Beijnum wrote:
>> through NATs and firewalls, it remains incompatible with PEPs or  
>> TCP proxies that need to split or monitor TCP's congestion control  
>> loop,
>
> Isn't that a feature?

You mean in that IPSEC effectively enforces the "end-to-endness" of  
TCP?  The fact that IPSEC prevents devices in the middle of the  
network from interfering with TCP's end-to-end _reliability_ and data  
integrity is indeed a feature.  The fact that IPSEC also prevents  
devices in the network from implementing reasonable performance  
optimizations as flows cross specific network technologies or specific  
administrative domains is not a feature.  The end-to-end principle is  
about reliability, not performance; it has always been about  
reliability and not performance; the E2E paper even explicitly pushes  
the point that in-network mechanisms can be useful and valuable to  
enhance performance (they were talking about in-network retransmission  
in particular) but just aren't a substitute for end-to-end reliability  
mechanisms.  I even asked Dave Clark about his perspective on this  
when I ran into him a few weeks ago, and he confirmed this  
interpretation of the E2E paper.  The idea that the end-to-end  
principle might or should also apply to performance-related mechanisms  
such as congestion control came much later and IMO is a complete  
misapplication of the principle.  There are reasonable technical  
arguments for (and against) end-to-end congestion control versus  
sectioned or hop-by-hop CC, of course, but those arguments have  
nothing to do with the end-to-end principle as proposed in the  
original E2E paper.

> Isnt't that basically what SSL/TLS does today?

Yes, with the additional difference (unimportant prorocol-wise but  
very important in practical use) that IPSEC is generally deployed and  
administered at OS level whereas SSL is deployed and administered by  
individual applications.  SSL also needs to be modified/redesigned for  
each transport (e.g. DTLS for UDP) whereas IPSEC is in theory  
transport-neutral (although the recent NAT traversal enhancements  
defeat this advantage).  The general point I was trying to make was  
that if we want IPSEC-style, application-transparent, transport- 
neutral security in some form, the best place for it to happen is  
above the Flow Layer (congestion control, PEPs, etc.) but below  
whatever protocol provides the application-visible transport semantics  
(reliability, ordering, etc.)

That said, you make an excellent point - since SSL over TCP is already  
ubiquitous and the convention of deploying SSL at application rather  
than system level is merely a convention and by no means essential to  
the protocol...   If we wanted to provide system-level, application- 
transparent security for MPTCP (or CMT-SCTP or SST or...) while  
providing as much backward compatibility as possible, maybe the right  
way to do that is to layer MPTCP over SSL over TCP.  That way secured  
MPTCP flows would look to the network (and to middleboxes in the  
network) indistinguishable from, say, ordinary HTTPS flows, which may  
be a really good thing.

> are being attacked). So TLS may suffice.

Yes, very true.

> frame/packet (not sure what the right terminology is) over the same  

I think we're thinking along the same lines here.

Bryan

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 10:24:06 +0200

On May 18, 2009, at 5:32 PM, Ford, Alan wrote:
>> protocol

Yes, this is exactly the type of thing I had in mind.

> appropriate than using TLS? (Aside from being transparent to the

Yes; see my last response to Iljitsch, speculating that MPTCP-over-TLS- 
over-TCP might actually provide the best combination of protocol  
properties (though with the slight pragmatic challenge that we'd  
probably need to develop OS kernel-level implementations of SSL/TLS...).

Bryan

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 19 May 2009 10:16:46 +0100

I It seems nice to layer MTCP over TLS/TCP, but I see two issues:
1. We need at least one endpoint to have a verifiable certificate.
2. The public key operations involved seem a bt heavyweight to be  
used in *every* TCP connection.
Perhaps a better fit is Mark's recent work on TCP crypt, which is an  
opportunistic encryption layer over tcp that doesn't require  
certificates and is a bit lighter in terms of public key ops..

That said, chunking the data and introducing the sequence numbers in  
the payload seems worthy of further exploration.
Costin

On 19 May 2009, at 09:24, Bryan Ford wrote:

> Yes; see my last response to Iljitsch, speculating that MPTCP-over- 
> TLS-over-TCP might actually provide the best combination of  
> protocol properties (though with the slight pragmatic challenge  
> that we'd probably need to develop OS kernel-level implementations  
> of SSL/TLS...).
>
> Bryan
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 19 May 2009 12:25:52 +0200

> TCP?

Yes.

> The fact that IPSEC also prevents devices in the network from  
> implementing reasonable performance optimizations as flows cross  
> specific network technologies or specific administrative domains is  
> not a feature.

I wouldn't want attackers to be able to slow my traffic down to a crawl.

But can you explain a bit more about these benevolent optimizers? What  
is it that they do, exactly?

> That said, you make an excellent point - since SSL over TCP is  
> already ubiquitous and the convention of deploying SSL at  
> application rather than system level is merely a convention and by  
> no means essential to the protocol...

I'd say that proper layering would be to provide authentication below  
the TCP layer and encryption above it, as encryption benefits from  
TCP's ordering, and TCP benefits from IPsec's authentication.

> If we wanted to provide system-level, application-transparent  
> security for MPTCP (or CMT-SCTP or SST or...) while providing as  
> much backward compatibility as possible, maybe the right way to do  
> that is to layer MPTCP over SSL over TCP.  That way secured MPTCP  
> flows would look to the network (and to middleboxes in the network)  
> indistinguishable from, say, ordinary HTTPS flows, which may be a  
> really good thing.

Well, it would look like that even if TLS is multiplexed over multiple  
subflows, but:

>> An interesting approach would be TCP over TLS over TCP where the  
>> TLS protects against the issues with middleboxes rewriting data  
>> that we talked about. To do this well, it would be helpful to send  
>> each TLS frame/packet (not sure what the right terminology is) over  
>> the same path.

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 14:06:53 +0200

> 1. We need at least one endpoint to have a verifiable certificate.

IPSEC (and any other secure protocol with strong authentication) has  
the same issue, no matter where in the stack it's layered.  If you  
want strong authentication, you have to have a strong identity to  
authenticate.  Although a self-certifying host namespace as in HIP can  
help by making the name effectivity double as its own verifiable  
certificate.  I don't know if anyone has designed a HIP-like self- 
certifying namespace for SSL/TLS, but it could certainly be done.

> used in *every* TCP connection.

True - but even a lot of the stuff TCP traditionally does, such as  
establishing a congestion control context, is "a bit heavyweight" to  
be done once for *every* application stream, as we discovered  
particularly acutely in the context of HTTP/1.0.  Because TCP has  
traditionally been doing all this heavyweight stuff once for every  
application stream, applications have traditionally been forced to  
compensate - e.g., as HTTP/1.1 did by adding persistent/pipelined  
connections, even though persistent/pipelined connections introduce a  
bunch of problems of their own like extra compatibility and  
unnecessary serialization of independent transactions.  The real  
solution is to modify the transport to stop doing all the heavyweight  
stuff for every application stream, instead doing the heavyweight  
stuff once to establish a secure, congestion controlled session  
between a host pair and then allowing the application to open many  
lightweight streams that reuse the same heavyweight communication  
state.  That was one of the main ideas of my experimental Structured  
Stream Transport (http://brynosaurus.com/pub/net/sst-abs.html), but it  
shouldn't be difficult to integrate many of the same ideas in a "next- 
generation TCP" that retains a stronger emphasis on backward  
compatibility.

> certificates and is a bit lighter in terms of public key ops..

This also sounds reasonable to me, but it basically just sounds like a  
different "control plane" key exchange scheme that could be attached  
to one basic "data plane" encryption/authentication protocol.  There  
are at least three such identity/key exchange schemes that seem useful  
and relevant in different systems: "weak identities", ala TCP crypt;  
PKI-based identities, as in SSL/TLS/IPSEC; and self-certifying  
identities, as in HIP or UIA.  Why should we necessarily have to  
choose one approach?  They should be pluggable modules configurable/ 
selectable by system administrators or applications.  At any rate,  
this is all security-land stuff, and I wouldn't really expect that to  
be in the scope of this BOF/WG other than to the point of considering  
how security layers like SSL/TLS/IPSEC should fit into a multipath- 
capable next-generation TCP architecture.

Bryan

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 15:01:36 +0200

> I wouldn't want attackers to be able to slow my traffic down to a  

On-path attackers can by definition do anything to your traffic,  
including cut off connectivity completely, so I assume you're  
referring only to off-path attackers here.  Defense against such off- 
path attacks against performance is indeed important, but that doesn't  
imply that such defenses must operate by denying _anyone_ in the  
network - including on-path elements - the ability to see or affect  
performance-related aspects of flows.  That's one way to provide such  
a defense, but it amounts to throwing out the baby with the bathwater.

Routing faces the same security issues: without appropriate security  
measures off-path attackers can potentially capture other hosts'  
traffic by injecting BGP announcements for prefixes they don't own, or  
dynamically impersonate other hosts by source address forgery.  One  
conceivable "end-to-end-style" solution would be to move the Internet  
to authenticate source routing, where end hosts are responsible for  
determining the complete route a packet is to take, and embedding that  
route in each packet along with a series of verifiable authentication  
tokens - but no one is seriously considering such an approach to  
dealing with the Internet's routing vulnerabilities, in part because  
it takes too much control away from in-network entities (ISPs,  
corporate network admins) who have legitimate needs to control how  
traffic flows on their network.  I think exactly the same applies to  
congestion control - preventing anyone in the network from having any  
access at all is one way of "protecting" congestion control, but it's  
far too heavyhanded.

> But can you explain a bit more about these benevolent optimizers?  
> What is it that they do, exactly?

They sit in the path and try to make TCP performance suck less, e.g.,  
when a mobile network link has a nontrivial non-congestion loss rate,  
or when a cellular link cuts out for a short period and causes TCP's  
congestion control to back off to zero unnecessarily when it comes  
back, or when a link has a high bandwidth-delay product causing TCP  
take forever to ramp up...  Many companies, including Cisco (look up  
their RBSCP documentation) already sell PEPs of various kinds; a lot  
of these devices are admittedly horrible hacks of one sort or another,  
but they clearly fill a market demand in the same way that NATs and  
firewalls do.  RFC3515 discusses various goals and techniques such  
devices commonly use, as well as the problems they typically cause  
(including breaking the end-to-endness of TCP and being completely  
incompatible with IPSEC).  For some relevant research papers on the  
principles these devices were built from, see for example Bakre's I- 
TCP (Indirect TCP), Kopparty's Split-TCP, Balakrishnan's TCP Snoop,  
Martin Swany's Logistical Session Layer (SC2004), and the various  
other references you'll find in section 3.1 of our "Breaking Up the  
Transport Logjam" paper.

> I'd say that proper layering would be to provide authentication  
> below the TCP layer and encryption above it, as encryption benefits  
> from TCP's ordering, and TCP benefits from IPsec's authentication.

Perhaps, as long as there remains some provision for in-path devices  
to get in the loop when they need to, ideally in a clean and  
composable fashion.

Bryan

> Well, it would look like that even if TLS is multiplexed over  
> multiple subflows, but:
>
>>> each TLS frame/packet (not sure what the right terminology is)  
>>> over the same path.
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 19 May 2009 15:27:49 +0200

> They sit in the path and try to make TCP performance suck less,  
> e.g., when a mobile network link has a nontrivial non-congestion  
> loss rate,

And how do they do that? By magically reconstituting lost packets?

A loss is a loss, congestion control is going to react to that no  
matter what.

> or when a link has a high bandwidth-delay product causing TCP take  
> forever to ramp up...

You mean like sending ACKs for data that hasn't been received yet?

Are there still boxes doing this? It's a terrible hack, it breaks  
every session where a packet gets lost.

> Many companies, including Cisco (look up their RBSCP documentation)  
> already sell PEPs of various kinds;

A dubious endorsement if I ever heard one...

The network should stick to dropping packets and setting the ECN bits,  
and not second guess TCP congestion control. That way, we'll be stuck  
with NewReno for the rest of eternity like NATs force us to stay with  
TCP and UDP.

> RFC3515 discusses various goals and techniques such devices commonly  
> use

The SIP Refer Method ?

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 06:40:07 -0700

> And how do they do that? By magically reconstituting lost packets?

Generally by spoofing the endpoint, i.e., by acting like a proxy, even
though the source thinks it's talking to the endpoint, e.g.:
	- sending spoofed ACKs to the source
	- buffering data and/or re-framing the segments
	- adjusting option values (e.g., faking a smaller RTT)

On wireless links, they can reduce the non-congestion loss by doing
"FEC" - either passively (sending the same segment multiple times, using
true FEC mechanisms) or actively (buffering and retransmitting until
received at some upstream optimizer).

> A loss is a loss, congestion control is going to react to that no matter
> what.

Not if it doesn't see the loss anymore. But it would see, e.g., much
larger RTTs, phony smaller RTTs, reordering, and/or highly variable delays.

> Are there still boxes doing this? It's a terrible hack, it breaks every
> session where a packet gets lost.

Yes, there are. It doesn't have to break during loss if the spoofer
holds the data until it gets the upstream ACK - i.e., if it emulates a
proxy. It's still evil (use a nontransparent proxy if that's what you
want), though.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 06:42:38 -0700

>> when a mobile network link has a nontrivial non-congestion loss rate,
...
> with NewReno for the rest of eternity like NATs force us to stay with

One PS -

a PEP that doesn't spoof ACKs can still reduce link errors by doing FEC
or ARQ; that PEP is just a software link conditioner.

Yes, it messes with TCP - esp. the ARQ that effectively makes the PEP
look like a variable delay link - but it's not much different than what
is already done at the link layer. The evil ones spoof the endpoint; the
"kosher" ones aren't evil, but can impact TCP too.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 19 May 2009 16:07:00 +0200

On 19 mei 2009, at 15:40, Joe Touch wrote:

>> matter

> Not if it doesn't see the loss anymore.

Well, either the packet is lost or it isn't.  :-)

If someone in the middle buffers then it's not really lost, just a  
very particular instance of duplication in the network...

> proxy.

Who wants to keep that much state?

Ideally, there would be a TCP proxy in the middle of the ocean or in  
the satellite.

The "right" way to do this statelessly would be to spoof a SACK, which  
is renegotiable.

My netstat -s tells me:

         306555606 packets received
                 171992750 acks (for 3031050254 bytes)
                 7564350 duplicate acks
                 31162 acks for unsent data

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 09:19:30 -0700

Iljitsch van Beijnum wrote:
> If someone in the middle buffers then it's not really lost, just a very
> particular instance of duplication in the network...

Sometimes the duplication is removed too, FWIW.

> Who wants to keep that much state?

People who make money selling this sort of stuff ;-)

> Ideally, there would be a TCP proxy in the middle of the ocean or in the
> satellite.

Rarely; more typically, the "PEP" is implemented at the endpoints of a
subpath that traverse the oceanic/satellite path. Satellites and
undersea cables tend to be quite dumb.

> The "right" way to do this statelessly would be to spoof a SACK, which

I don't think it's "right" to spoof anythhing. ;-)

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 11:35:11 -0700

Bryan Ford wrote:
...
> including cut off connectivity completely, so I assume you're referring
> only to off-path attackers here.  Defense against such off-path attacks
> against performance is indeed important, but that doesn't imply that
> such defenses must operate by denying _anyone_ in the network -
> including on-path elements - the ability to see or affect
> performance-related aspects of flows.  ...

Well, I agree with "see" the aspects.

I disagree with allowing anyone other than endpoints to muck with the
endpoint aspects - i.e., presenting fictitous RTTs, window sizes, or ACKs.

IPsec and TCP MD5 (and its successor, TCP-AO) are intended to prevent
exactly that interference. They are deployed where the endpoints *want*
that sort of e2e protection. Sure, you can put the result inside TCP,
but then your middlebox is playing with the outer ('link') TCP, not the
e2e one, which is fine.

> when a mobile network link has a nontrivial non-congestion loss rate, or
> when a cellular link cuts out for a short period and causes TCP's
> congestion control to back off to zero unnecessarily when it comes back,
> forever to ramp up...  Many companies, including Cisco (look up their
> RBSCP documentation) already sell PEPs of various kinds; a lot of these
> devices are admittedly horrible hacks of one sort or another, but they
> clearly fill a market demand in the same way that NATs and firewalls
> do.  RFC3515 discusses various goals and techniques such devices
> commonly use, as well as the problems they typically cause (including
> breaking the end-to-endness of TCP and being completely incompatible
> with IPSEC).  For some relevant research papers on the principles these
> devices were built from, see for example Bakre's I-TCP (Indirect TCP),
> Kopparty's Split-TCP, Balakrishnan's TCP Snoop, Martin Swany's
> Logistical Session Layer (SC2004), and the various other references
> you'll find in section 3.1 of our "Breaking Up the Transport Logjam" paper.

PEPs are fine - they're just a virtual link layer. Devices that modify
TCP headers, spoof ACKs, or otherwise wouldn't work when the segments
are authenticated shouldn't work anyway, IMO.

>>> already ubiquitous and the convention of deploying SSL at application
>>> rather than system level is merely a convention and by no means
>>> essential to the protocol...
>>
> Perhaps, as long as there remains some provision for in-path devices to
> get in the loop when they need to, ideally in a clean and composable
> fashion.

Who decides when a device "needs" to? IMO, when the endpoint
authenticates or encrypts, it is already saying the middlebox neither
needs nor is allowed to do this sort of thing...

Joe

---

From: Bryan Ford <[email-masked]>
Date: Tue, 19 May 2009 20:39:35 +0200

> proxy. It's still evil (use a nontransparent proxy if that's what you

There are various flavors of evilness, too, not necessarily all on one  
continuum.

For example, TCP splitting approaches (i.e., outright interposition)  
are evil because they break TCP's end-to-endness by putting hard state  
into the network: e.g., if the proxy goes down the connection dies.   
But at least they are more or less cleanly composable: if a  
communication path crosses two "interesting" links, each of which has  
a pair of TCP splitting PEPs on either end using the above tricks to  
make performance not suck, then there's a decent change the TCP  
connection crossing both pairs of PEPs will behave in a reasonable  
fashion and not suck.  That's because TCP splitting effectively  
divides the path into several independent congestion control loops, so  
that the two path sections crossing the "interesting" links are  
isolated from each other and the PEPs around their links can do the  
appropriate magic independent of what's going on in the rest of the  
path.

On the other hand, other approaches are evil for different reasons.   
For example, in one approach PEPs use to get the TCP sender to ramp up  
faster and fill up a high-bandwidth-delay-product link, known as TCP  
splitting, the PEP basically splits an each ACK coming down the return  
path for a given segment (e.g., bytes 1000-1999) into ACKs for several  
smaller segments (e.g., 1000-1249, 1250-499, 1500-1749, 1750-1999), in  
order to trick the TCP sender into increasing its cwnd several times  
per segment instead of just one.  This trick is "not evil" in the end- 
to-endness sense because it doesn't do anything bad to connection  
reliability or fate sharing: no data is lost if anything goes wrong,  
and it doesn't even inherently place any hard state in the network  
(although ACK splitting PEPs usually maintain hard state for other  
reasons).  On the other hand, this trick is very evil in at least two  
other ways.  First, its behavior under composition can be nasty: for  
example, if a path happens to cross two high-bandwidth-delay links  
with ACK splitting PEPs on their endpoints, each one amplifying TCP's  
aggressiveness, then the combination amplifies TCP's aggressiveness by  
16x.  Oops.  Second, the same technique can be used as an attack by  
malicious or greedy clients on servers, to gain an unfair share of the  
server's bandwidth.  Hence, we see one ongoing line of research in all  
the benefits and wonderful ways to use ACK splitting (e.g., Jin,  
"SPACK: rapid recovery...", '99; Hasegawa, "Receiver-based ACK  
Splitting...", '07), together with a concurrent line of research on  
ways to shut down the use of such techniques (e.g., Savage, "TCP  
congestion control with a misbehaving receiver", 99; Stanojevic, "Can  
you fool me? ...", 08).  What fun!

In our "Breaking the Logjam" architecture we take the position that  
when evil in-network performance enhancement tricks prove necessary,  
they should be done via the former, cleanly-composable, interposition- 
based approaches - but only in the Flow Layer, where end-to-end  
reliability and fate sharing isn't an issue because all Flow Layer  
state can be reconstructed on demand by the higher-level Transport  
Layer.  Building MPTCP or SCTP++ or whatever atop multiple legacy TCP  
flows would be consistent with this architecture, effectively removing  
the primary reason TCP-splitting PEPs are evil.  Whether such PEPs are  
still evil for other reasons depends on one's perspective and on the  
specific PEP's design. :)

Bryan

>
> Joe
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkoStrYACgkQE5f5cImnZrteOwCfemNuw2XMVr+ij+ub+MQCbtIZ
> S9gAnR/68I7Cs2yBiYJT4qmTbnzPyKmB
> =Rp3+
> -----END PGP SIGNATURE-----

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 11:49:34 -0700

Bryan Ford wrote:
> For example, TCP splitting approaches (i.e., outright interposition) are
> evil because they break TCP's end-to-endness by putting hard state into
> the network: e.g., if the proxy goes down the connection dies.  But at
> least they are more or less cleanly composable: if a communication path
> crosses two "interesting" links, each of which has a pair of TCP
> splitting PEPs on either end using the above tricks to make performance
> not suck, then there's a decent change the TCP connection crossing both
> pairs of PEPs will behave in a reasonable fashion and not suck.  That's
> because TCP splitting effectively divides the path into several
> independent congestion control loops, so that the two path sections
> crossing the "interesting" links are isolated from each other and the
> PEPs around their links can do the appropriate magic independent of
> what's going on in the rest of the path.

As I said, a PEP that emulates a link is fine. Anything that spoofs a
TCP segment or modifies a TCP header is not.

To that end, making the TCP info visible to the PEP can make a PEP's
life easier, though it's still possible to install a PEP over a flakey
or non-congestion lossy link and have a benefit even for completely
encrypted traffic.

...
> In our "Breaking the Logjam" architecture we take the position that when
> evil in-network performance enhancement tricks prove necessary, they
> should be done via the former, cleanly-composable, interposition-based
> approaches - but only in the Flow Layer, where end-to-end reliability
> and fate sharing isn't an issue because all Flow Layer state can be
> reconstructed on demand by the higher-level Transport Layer.

If by "cleanly composeable" you mean "doesn't spoof, and doesn't
modify", then I agree. I can't see how you can assert anything about a
'flow layer' that isn't already true about TCP itself - i.e., TCP is a
flow, and you shouldn't be interfering with its endpoints.

> Building
> MPTCP or SCTP++ or whatever atop multiple legacy TCP flows would be
> consistent with this architecture, effectively removing the primary
> reason TCP-splitting PEPs are evil.  Whether such PEPs are still evil
> for other reasons depends on one's perspective and on the specific PEP's
> design. :)

So if the PEP doesn't interfere with your transport (MPTCP, SCTP++),
it's OK. But if it interferes with an underlying transport (TCP), that's
fine, right? Sounds very NIMBY to me...

IMO, if you aren't prepared to have a PEP muck with your transport, you
shouldn't endorse it mucking with the transport of others, period.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 19 May 2009 20:55:09 +0200

On 19 mei 2009, at 20:39, Bryan Ford wrote:

> in one approach PEPs use to get the TCP sender to ramp up faster and  
> fill up a high-bandwidth-delay-product link, known as TCP splitting,  
> the PEP basically splits an each ACK coming down the return path for  
> a given segment (e.g., bytes 1000-1999) into ACKs for several  
> smaller segments (e.g., 1000-1249, 1250-499, 1500-1749, 1750-1999),  
> in order to trick the TCP sender into increasing its cwnd several  
> times per segment instead of just one.

Does this work across a wide variety of implementations?

My understanding of RFC 2581 is that an implementation shouldn't react  
to the _number_ of ACKs but rather to how much data is ACKed.

And in the pseudocode of my draft I basically implement a SACK fast  
retransmit based on the amount of out-of-order data SACKed because the  
number of ACKs quickly becomes meaningless in a one-ended MPTCP  
because there will be cross-ACKs between the subflows (one paper,  
forget which, actually cites this as an important performance  
enhancing feature) and also a lot of duplicate ACKs.

I previously said that the "correct" way to trick a TCP sender into  
ramping up would be with SACK, but then again I have no idea what an  
implementation would do when it sees data following the cumulative ACK  
SACKed without a gap...

---

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 19 May 2009 17:59:48 -0400

Joe,

> flow, and you shouldn't be interfering with its endpoints.

Just to clarify:  our proposal factors out the congestion control functionality out of the transport layer into a flow layer.  Our idea is that PEPs do not have to spoof anything if the ends are willing to negotiate with them in the middle.  Flow layer congestion control loops do not have to go end-to-end; they go end-middle-middle-...-end, with as many flow layer middleboxes as you want to have in the end-to-end loop.  

Keep in mind that we do not really care about what congestion control tricks any PEPs may do, we care that they muck with TCP end-to-end semantics.

In our architecture, "hard" state (reliability) and stuff of application semantics (the API) are retained in the transport layer, end-to-end.  The middleboxes/PEPs can do performance tuning at the flow layer, and do not have to retain any hard state.  If a PEP drops off, our end-to-end TCP connection can re-sync over a new/different flow layer instance.

> fine, right? Sounds very NIMBY to me...

What our architecture does is makes it so that PEPs can actively engage with the ends at the flow layer (using TCP or DCCP or whathaveyou), and the transports can remain disengaged from this conversation.  Again, the PEPs care about congestion control and performance; they don't care about transport semantics.  I don't see why they would want to muck in transport layer semantics (acks and reliability) if we can let them cleanly optimize for performance.

- jana

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2009 15:10:32 -0700

...
> Just to clarify:  our proposal factors out the congestion control
> functionality out of the transport layer into a flow layer.

TCP doesn't factor them out; they're integrated.

> Our idea is
> that PEPs do not have to spoof anything if the ends are willing to
> negotiate with them in the middle. 

Yes. But TCP isn't so willing.

> Flow layer congestion control loops
> do not have to go end-to-end; they go end-middle-middle-...-end, with as
> many flow layer middleboxes as you want to have in the end-to-end loop. 

Well, you're saying that hop-by-hop congestion control is sufficient,
rather than E2E. You can have that debate with Clark/Reed/Saltzer.

> Keep in mind that we do not really care about what congestion control
> tricks any PEPs may do, we care that they muck with TCP end-to-end
> semantics.

I would hope that you care that they do NOT muck with TCP E2E semantics.
The point is that TCP E2E semantics includes E2E flow control.

> In our architecture, "hard" state (reliability) and stuff of application
> semantics (the API) are retained in the transport layer, end-to-end. 
> The middleboxes/PEPs can do performance tuning at the flow layer, and do
> not have to retain any hard state.  If a PEP drops off, our end-to-end
> TCP connection can re-sync over a new/different flow layer instance.

It sounds like your TCP isn't our TCP.

> Thinking of the multipath discussion we are having on this list, MPTCP
> (with some tweaks) would nicely fit our architecture's new "transport"
> layer, and TCP would nicely fit our new "flow" layer. 

You say "transport", I say "transport".

You say "flow", I say "hop".

I.e., it seems to me you're using TCP as a link layer (been talking to
the DTN folks? ;-)

...
> What our architecture does is makes it so that PEPs can actively engage
> with the ends at the flow layer (using TCP or DCCP or whathaveyou), and
> the transports can remain disengaged from this conversation.  Again, the
> PEPs care about congestion control and performance; they don't care
> about transport semantics.  I don't see why they would want to muck in
> transport layer semantics (acks and reliability) if we can let them
> cleanly optimize for performance.

Right - but the issue is whether when you see TCP you can know it's just
a hop version of the E2E TCP. If you can't tell the difference, you
shouldn't be playing with it, since that TCP might just be the transport
layer you said you didn't want to bother.

Joe

---

From: Janardhan Iyengar <[email-masked]>
Date: Tue, 19 May 2009 18:18:14 -0400

Iljitsch,

> to the _number_ of ACKs but rather to how much data is ACKed.

RFC 2581 allows it (see Savage's paper that Bryan mentioned). But yes, Appropriate Byte Counting (ABC, RFC 3465) fixes it to actually count the number of bytes acked...  I don't know the deployment status of ABC (I think there may have been a study that looked at several things including ABC deployment on webservers...)

> number of ACKs quickly becomes meaningless in a one-ended MPTCP because 
> there will be cross-ACKs between the subflows (one paper, forget which, 
> actually cites this as an important performance enhancing feature) and 
> also a lot of duplicate ACKs.
> 

Absolutely.  I think that is necessary to do because the MPTCP sender is *causing* the reordering (by splitting the flow).  And I don't know if you can at all do MPTCP without SACK.

I'll also note that yes, the number of dupacks increases as well with interesting side-effects... one simply that the number of packets on the return path will increase (delayed-acks are sent roughly once every two packets, and dupacks are sent for every packet received out of order).  We use an algorithm in CMT to reduce the number of dupacks without affecting the number of dupacks required for fast-retransmissions.  We do use a bit in the header however to do this.

> implementation would do when it sees data following the cumulative ACK 

Which is exactly why the correct way to trick a TCP sender into ramping up would be to not do it. :-)

Tricks exploit protocol "gullibilities", and as much interest and work that goes into identifying these gullibilities and exploiting them for in-loop performance benefits, that much interest and work goes into identifying and fixing these gullibilities.  Interestingly, neither of these groups is trying to actively compete with the other, and yet there is this arms-race!

- jana

---

From: <[email-masked]>
Date: Wed, 20 May 2009 07:28:58 +0100

> Well, you're saying that hop-by-hop congestion control is sufficient,

No, he's not. the middleboxes have a view of two different subnets.
And he's still mentioning an end-to-end loop.

> rather than E2E. You can have that debate with Clark/Reed/Saltzer.

...should Clark/Reed/Saltzer ever write a treatise on congestion control.

Their end-to-end papers are about reliability and performance,
not about the somewhat different congestion control, fairness and
competition.

Rhetoric much, Joe?

L.

<http://www.ee.surrey.ac.uk/Personal/L.Wood/><[email-masked]>

...

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 20 May 2009 07:47:28 -0700

> And he's still mentioning an end-to-end loop.

Not for "flows"; besides, you basically agree that HBH is what he's
doing below. Why are you disagreeing here?

> not about the somewhat different congestion control, fairness and

Their paper is about the composition of services. If you have an
argument that some services are indeed composeable end-to-end, the
please make that case in detail.

Merely asserting that C/R/S doesn't apply for these services is not an
argument; it is an (unsubstantiated) assertion.

Joe

---

From: Janardhan Iyengar <[email-masked]>
Date: Wed, 20 May 2009 13:20:28 -0400

Joe,

> Yes. But TCP isn't so willing.

Yes, that is true, and TCP isn't so willing to do multipath either.  But that is what we are trying to do here.  Along the same lines, TCP isn't so willing to talk to PEPs right now, and we think that should change.

To be clear, we are talking about factoring congestion control out of the transport into a "Flow Layer" that interacts with middleboxes, and leave the rest of the stuff that applications care about (relaibility, order preservation, flow control, etc.) at the ends in the end-to-end transport layer (or as we call it, the "Semantic" Layer.)  So, we are saying that TCP has both of these functionalities, and we are splitting them into different layers.

More details at: "Breaking Up The Transport Logjam", at http://web1.fandm.edu/jiyengar/papers/layers-hotnets2008.pdf

> rather than E2E. You can have that debate with Clark/Reed/Saltzer.

Saltzer/Reed/Clark do not talk about congestion control, they talk only about reliability.  Noel Chiappa has a nice article called "Will The Real "End-End Principle" Please Stand Up?" (http://mercury.lcs.mit.edu/~jnc/tech/end_end.html), which you may already seen, and it should help clarify this issue.

Note that we are not pushing reliability into the network, only the performance-related function of congestion control.

This is in stark contrast with the PEPs that are in operation today and openly flout the e2e principle by "transparently" interposing on the e2e path.  We are,  proposing to pull our collective head out of the sand and see these PEPs, understand why they exist and why anyone cares to spend the thousands of dollars to make and/or buy them, and let the ends actively negotiate with them.  PEPs only care about performance (hence the subject of this thread), which is not a bad thing for anyone!  It becomes bad because our transports (TCP) conflate end-to-end, application-facing semantics---in particular, reliability---with the performance-related functions.  And for PEPs to do their performance tweaks, they must figure out how to get around TCP semantics.  We propose to take TCP semantics out of the PEP loop, into an end-to-end loop, and let the PEPs optimize away.

MPTCP is very much like our end-to-end "Semantic" Layer, and the TCP that they want to use is somewhat like our "Flow Layer", with the difference that it does not actively negotiate with the PEPs. But it is designed with the idea that the PEPs will do their optimizations, while e2e reliability will happen at the MPTCP level with DSNs and other such.

Let me note that the TCP in the two-sided mulipath draft is not like the TCP we know either; for one, use of the re-sync message changes completely the reliability expectations you would have from TCP.  When you start doing retransmissions on paths different from that of the original transmission (which you will have to do to manage path failure), you will have to change the TCP state machine.

> The point is that TCP E2E semantics includes E2E flow control.

Yes, but we are talking about congestion control, not flow control. Flow control continues to be at the ends.

> It sounds like your TCP isn't our TCP.

That is probably right... our "Semantic Layer" (what I referred to, admittedly ambiguously, as TCP) does not do congestion control, much like MPTCP.  Our Flow Layer interacts with the middleboxes and does congestion control.

> layer you said you didn't want to bother.

I'm not sure I understand this point.  For middleboxes to actively participate, there would need to be explicit negotiation of Flow (congestion control) instances on the end-to-end path, and that is how a PEP would be able to tell the difference.  Or did you mean something else?

- jana

---

From: Joe Touch <[email-masked]>
Date: Wed, 20 May 2009 10:33:08 -0700

Janardhan Iyengar wrote:
> Yes, that is true, and TCP isn't so willing to do multipath either.  But
> that is what we are trying to do here.  Along the same lines, TCP isn't
> so willing to talk to PEPs right now, and we think that should change.
> 
> To be clear, we are talking about factoring congestion control out of
> the transport into a "Flow Layer" that interacts with middleboxes, and
> leave the rest of the stuff that applications care about (relaibility,
> order preservation, flow control, etc.) at the ends in the end-to-end
> transport layer (or as we call it, the "Semantic" Layer.)  So, we are
> saying that TCP has both of these functionalities, and we are splitting
> them into different layers.
> 
> More details at: "Breaking Up The Transport Logjam", at
> http://web1.fandm.edu/jiyengar/papers/layers-hotnets2008.pdf
...
> I'm not sure I understand this point.  For middleboxes to actively
> participate, there would need to be explicit negotiation of Flow
> (congestion control) instances on the end-to-end path, and that is how a
> PEP would be able to tell the difference.  Or did you mean something else?

Let's put it this way. First show me a system that makes this work and
talks to middleboxes. Then we can talk about whether it makes sense to
call it TCP or not.

AFAICT, the problem with middleboxes is that you want to play nice with
them, but they don't want to play nice with anybody. They want to work
whether you want them there or not, whether you talk to them or not,
etc. That is nonsensical, and it's impossible to understand how to make
a protocol that expects to interact with a party that deliberately won't
play by the rules.

I.e., a middlebox that wants to play nice is, IMO, called a proxy. A
proxy wants you to know its there, and won't play if you don't. Proxies
work fine for doing the hop-by-hop stuff you're talking about.
Everything else is "making laws for the lawless", and (IMO) has been a
consistent waste of energy in the IETF.

Joe

---

From: <[email-masked]>
Date: Wed, 20 May 2009 18:48:59 +0100

Joe Touch said:
> Lloyd Wood said:
>> Joe Touch said
> Their paper is about the composition of services.

...despite, oddly, not mentioning services.

L.

DTN work: http://info.ee.surrey.ac.uk/Personal/L.Wood/saratoga/

<http://info.ee.surrey.ac.uk/Personal/L.Wood/><[email-masked]>

---

From: Joe Touch <[email-masked]>
Date: Wed, 20 May 2009 11:12:57 -0700

> ...despite, oddly, not mentioning services.

It doesn't use the word "composition" either.

I guess you've dismissed Baran's papers because they don't mention
packets, too?

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 21 May 2009 11:19:42 +0200

> More details at: "Breaking Up The Transport Logjam", at http://web1.fandm.edu/jiyengar/papers/layers-hotnets2008.pdf

Wow, that thing goes on forever, seemingly making the same points 50  
times...

Today, boxes in the middle of the network have two mechanisms to slow  
down flows, one of which is in wide use and the other isn't: dropping  
packets and ECN.

What we don't have is a mechanism to make flows ramp up faster than  
they otherwise would. Too bad we're basically using 2 bits for ECN to  
provide a 1-bit function, otherwise we could have done this with ECN,  
too.

But a better way would be to have a few more bits so a router can tell  
a flow how fast it may go, based on the available capacity at the  
router. That way, a flow reaches the optimal speed after a roundtrip.

It's such a shame that the IPv4 and especially IPv6 IP option  
mechanisms are so clunky that they're as good as unusable to implement  
something like this.

Ok, maybe it's time to get back to multipath now...