Subject: [multipathtcp] Initial ideas for API requirements

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 12 Nov 2009 03:33:37 +0100

Dear all,

draft-scharf-mptcp-api-00 contains a list of potential requirements on
the MPTCP API towards application. This list is of course just a
starting point. Currently, it is certainly not complete and may also
include requirements that will be removed in later stage, once the MPTCP
architecture spec evolves.

Still, we would really appreciate comments on the scope of the
(optional) extended API. Please find below a copy of the list in the
draft.

   REQ1:   Turn on/off MPTCP: An application should be able to request
           to turn on or turn off the usage of MPTCP.  This means that
           an application should be able to explicitly request the use
           of MPTCP if this is possible.  Applications should also be
           able to request not to enable MPTCP and to use regular TCP
           transport instead.  (This can be implicit in many cases,
           e.g., by the use of binding to a specific address versus all
           addresses).

   REQ2:   An application will want to be able to restrict MPTCP to
           binding to a given set of addresses or interfaces.

   REQ3:   An application should be able to know if multiple subflows
           are in use.

   REQ4:   An application should be able to extract a unique identifier
           for the connection (per endpoint), analogous to a port, i.e.
           it should be able to retrieve MPTCP's connection identifier.

   REQ5:   An application should be able to enumerate all subflows in
           use, obtain information on the addresses used by a subflow,
           and obtain a subflow's usage (e.g., ratio of traffic sent via
           this subflow).

   REQ6:   Set/get application profile, as discussed in the previous
           section.

   REQ7:   Constrain the maximum number of subflows to be used by an
           MPTCP connection.  (Or just infer from application profile?)

   REQ8:   Request a change in scheduling between subflows? (i.e. a more
           granular version of application profile?)

   REQ9:   Request a change in the number of subflows in use, thus
           triggering removal or addition of subflows.  (A finer control
           granularity would be: Request the establishment of a new
           subflow to a provided destination, and request the
           termination of a specified, existing subflow.)

   REQ10:  Control automatic establishment/termination of subflows?
           There could be different configurations of the path manager,
           e.g., 'try ASAP', 'wait until there is a bunch of data, etc.
           (Tied to application profile?)

   REQ11:  Set/get preferred subflows or subflow usage policies?  There
           could be different configurations of the multipath scheduler,
           e.g., 'all-or-nothing', 'overflow', etc.  (Again, tied to
           application profile).

   REQ12:  Set/get sporadic sending of segments on unused paths
           ("keepalives").

   REQ13:  An application should be able to modify the MPTCP
           configuration while communication is ongoing, i.e., after
           establishment of the MPTCP connection.

Thanks

Michael

---

From: Costin Raiciu <[email-masked]>
Date: Thu, 12 Nov 2009 16:02:00 +0900

How about redundancy? I can think of cases where the app might like  
to send all segments on all paths, to reduce end to end rtt, etc.

Costin

On 12 Nov 2009, at 11:33, SCHARF, Michael wrote:

> _______________________________________________
> multipathtcp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 12 Nov 2009 09:10:41 +0100

Michael,
> architecture spec evolves.

I am frightened by the long list of requirements and I fear that this
would make the API far too complex. The layering principle tells us that
we need to hide some of the details of the underlying layer to the upper
layer and experience should that this works well in practice.

To take an analogy, let's look at how the socket API evolved with
changes in TCP such as the introduction of congestion control, the fats
retransmit, the selective acks, timers computations, ... All of these
mechanisms have changed the behaviour of the TCP stack and its reaction
against losses but we never had to update the API to reflect this.

>            e.g., by the use of binding to a specific address versus all

I would love to have a higher layer API where the application indicates
that it needs a reliable delivery and the stack chooses the most
appropriate transport protocol (tcp, sctp, mtcp, ...) automatically.
This would allow the application and the transport to evolve independently.

>            binding to a given set of addresses or interfaces.

why ?

>    REQ3:   An application should be able to know if multiple subflows

why ? today's application do not ask whether there have been retransmissions

>            it should be able to retrieve MPTCP's connection identifier.

this looks reasonable

>            and obtain a subflow's usage (e.g., ratio of traffic sent via

why ? today's application do not ask whether there have been retransmissions

>    REQ6:   Set/get application profile, as discussed in the previous

remember that for intserv we almost never managed to have application
developers define their QoS requirements

>            MPTCP connection.  (Or just infer from application profile?)

the stack should do this automatically

>            granular version of application profile?)

why ? today's applications do not set for example the MSS to be used

>            termination of a specified, existing subflow.)

this is the job of the tcp implementation, not the application

>            (Tied to application profile?)

i don't see why an application should want to do this

>    REQ12:  Set/get sporadic sending of segments on unused paths

no

>            establishment of the MPTCP connection.

we should have as few mptcp configuration as possible, otherwise pmtcp
will be too complex for the application developers

Olivier

-- 
http://inl.info.ucl.ac.be , UCLouvain, Belgium

---

From: "Ford, Alan" <[email-masked]>
Date: Thu, 12 Nov 2009 09:57:07 -0000

Hi Olivier,

Thanks for feedback. The purpose of this list of requirements is to
generate exactly this discussion :)

> layer and experience should that this works well in practice.

Agreed. However, there are two strands we need to consider when defining
this API:

1. Ensuring the same control as is provided for TCP is also provided for
MPTCP
2. Allowing an application to exploit some of the _key beneficial
features_ of MPTCP.

> against losses but we never had to update the API to reflect this.

A good analogy, yes.

> This would allow the application and the transport to evolve
independently.

There several issues here.

One is for a MPTCP-aware application to request the use of Multipath.
Pasi's suggestion of AF_INETMP or similar would seem a possibly solution
to this. The next is for a MPTCP-aware application to request NOT to use
MPTCP. Here, the use of AF_INET / AF_INET6 would not be sufficient,
since how would the API tell the difference between this and a
non-MPTCP-aware application? 

So we probably need another call for MPTCP-aware apps. Hmm. If we accept
this as a requirement, anyway.

> >            binding to a given set of addresses or interfaces.

Analogous to TCP (binding to a given address, or SO_BINDTODEVICE). Just
needs to be able to cope with >1 interface/address!

> why ? today's application do not ask whether there have been
retransmissions

Combining REQ3 & REQ5, probably all we need is the ability to get the
list of address pairs in use. This is analogous to getpeername() and
getsockname().

> developers define their QoS requirements

However, in MPTCP this does make a _real_ difference. This is not simply
a case of saying "do I want the best QoS or not the best QoS" (for which
there's no wonder an application always says they want the best). This
is a case of "do I want throughput, or reduced jitter" for example.

> the stack should do this automatically

Yep, and the profile could definitely influence this.

> will be too complex for the application developers

Agree to some extent. We don't want it so automatic, so inflexible, that
application developers (and of course users) don't find it sufficiently
beneficial to use.

Regards,
Alan

---

From: Zhen Cao <[email-masked]>
Date: Thu, 12 Nov 2009 19:09:20 +0800

> against losses but we never had to update the API to reflect this.

My concern is, if we update the stack to support new features like
mptcp, everything on the application level will be changed even if
they do not want to use mptcp.

> this is the job of the tcp implementation, not the application

Implementation can do the job, and if we export these implementation
to the application by a new set of APIs, developers will benefit  from
it.

Thanks,
-Zhen
China Mobile

---

From: William Herrin <[email-masked]>
Date: Thu, 12 Nov 2009 10:21:42 -0500

>           and obtain a subflow's usage (e.g., ratio of traffic sent via

Hi Michael,

Specific items the application may want to know:

1. Whether multipath was negotiated, regardless of the number of
subflows currently in use.

2. Both endpoint connection IDs if multipath is negotiated. Primarily
for logging and matching your logs to the remote or middlebox's logs.

3. Classic TCP connection IDs (2xIP,2xport) if multipath is not negotiated.

4. What are the local and remote candidate addresses for subflows? Not
just the ones in use, the ones that might be used.

Are 3 and 4 the same call that returns a variable size structure? Or
are they different calls that the app has to be aware of?

5. What are the actual subflows in use?

6. Subflow characteristics such as retransmitted packet count, total
packet count, delay characteristics.

Also, for each of these items, WHEN do you tell the application? Upon
connection startup only? Upon request by the application? With a
callback as they change? All of the above?

>   REQ12:  Set/get sporadic sending of segments on unused paths

Is there a reason to do this in the extended API instead of the
current API? Put another way, why would we want keepalives on only
chosen subflows instead of all of them?

>   REQ13:  An application should be able to modify the MPTCP

REQ14: multiply addressed connect. Should be able to pass a list of
addresses to connect() instead of just one so that MPTCP can implement
a reasonable degree of parallelism when establishing the initial
connection. The whole "wait for connect to address A to time out,
cancel it and then try address B" deal that applications do now would
be suboptimal in an MPTCP scenario.

REQ15: connect by destination hostname. Not only does this avoid
making every single application developer reinvent the wheel, it also
enables:
* MPTCP to re-find the destination out of band should all known
addresses fail while the connection is active.
* MPTCP apps to seamlessly handle future successor protocols to IPv6
by not requiring the app to understand their address formats.

And if this means apps actually honor the DNS TTL, that'd be a bonus.

REQ16: set source hostname. Same as 15 but allows the connection's
acceptor to re-find the initiator.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 12 Nov 2009 17:26:33 +0100

Alan,

> non-MPTCP-aware application? 

I would suggest to have mtcp on by default and an option to disable it
if required by the application. Most applications don't care of their
data is transported to the destination, the only issue for them is that
the data must reach the destination reliably.

> needs to be able to cope with >1 interface/address!

maybe this could be useful on some multihomed hosts

> list of address pairs in use. This is analogous to getpeername() and

i don't think that application needs more than this

> is a case of "do I want throughput, or reduced jitter" for example.

i'm not convinced that application developpers will want to deal with
these details. We need to keep the API as simple as possible (and
perhaps simpler than that)

> Yep, and the profile could definitely influence this.

For me, profile is similar to traffic contract in intserv, I don't
expect that many applications will define their profile. Most
applications will use the default profile and this what we should optimise

> application developers (and of course users) don't find it sufficiently

We want a multipath tcp that works for 99% of applications. Some
applications implement a transport protocol inside the application, but
these applications are very rare and we should not optimise mptcp for
the needs of such applications since anyway they will implement complex
stuff in the application layer

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 12 Nov 2009 17:31:17 +0100

Zhen,
> they do not want to use mptcp.

REQ0 should be that an existing application can benefit from MPTCP
without any change to its code.

If we force an application to change to support MPTCP, we'll have the
same problem to find applications as stcp and ipv6 in the past

> to the application by a new set of APIs, developers will benefit  from

We should maintain a good separation between the layers. We do not
expose to applications protocol issues such as MSS, retransmission
mechanisms, congestion control mechanism, ... For me, the number of
subflows is equivalent to the congestion control and the retransmission
mechanisms that are used by TCP today. These improve the performance,
but the service provided to the application is still a reliable byte
stream.

Olivier

---

From: João Taveira Araújo <[email-masked]>
Date: Fri, 13 Nov 2009 01:47:44 +0000

Olivier,

> same problem to find applications as stcp and ipv6 in the past

This is an oversimplification. MPTCP looks like standard TCP from the 
perspective of any middlebox, and therefore is inherently deployable. 
The same can't be said of sctp or ipv6.

The fork in the road seems to be that if you masquerade MPTCP as a 
standard TCP socket for legacy apps, you might break prevailing 
assumptions. In my mind there is no reason why apps which gain the most 
benefit (bulk data, mobility, redundancy) would be hesitant in upgrading 
once OS support is in place.

Joao.

---

From: Scott Brim <[email-masked]>
Date: Fri, 13 Nov 2009 11:12:07 +0900

> against losses but we never had to update the API to reflect this.

Olivier, at a meta-level we want to expose _more_ details of transport
to the application, not less.  Having Transport treat the application as
if it were a child is just like having the network treat the endpoint as
if it were a child.

Just because capabilities are available in an API doesn't mean every app
needs to use them.  We can have reasonable defaults, or "unspecified"
could mean that Transport should do what it thinks best.  This is
essentially what will happen with the old socket API in any case.

Regards ... Scott

---

From: Scott Brim <[email-masked]>
Date: Fri, 13 Nov 2009 11:14:13 +0900

> perspective of any middlebox, and therefore is inherently deployable.

Well, except for an extra option, and changes in higher layer packet
flow (some middleboxes might not like either of these).

swb

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 13 Nov 2009 09:37:22 +0100

William,
> subflows currently in use.

Today, applications do not require to know whether rfc1323, sack or
other TCP option
shave been negotiated on a given TCP connection.

> are they different calls that the app has to be aware of?

We need to distinguish between information that the application needs to
know
for each TCP connection and debugging information that we need. To take
a common
problem, consider PathMTU discovery. Some firewalls block ICMP messages
and this breaks
PathMTU discovery. These problems are debugged by the network
administrators without requiring
any change to application and without needing to extend the applications
to deal with these
problems.

> packet count, delay characteristics.

This is clearly something that the application does not need from an API.

> chosen subflows instead of all of them?

keepalives are handled by the transport protocol internally, they do not
need to be exposed to the applications

> by not requiring the app to understand their address formats.

This looks reasonable to me. I would suggest Stuart Cheshire's presentation
at IETF72as it contains useful info about API design issues
http://www.stuartcheshire.org/IETF72/

Olivier

---

From: William Herrin <[email-masked]>
Date: Fri, 13 Nov 2009 20:43:17 -0500

On Fri, Nov 13, 2009 at 3:37 AM, Olivier Bonaventure
<[email-masked]> wrote:
> other TCP options have been negotiated on a given TCP connection.

Hi Olivier,

Let's look at it a different way: is there a reason that an
application should be denied knowledge of whether multipath TCP was
negotiated on a particular connection? The stack certainly knows, so
it won't complicate any code if there's a getsockopt that reports
whether multipath is in use, will it?

> know for each TCP connection and debugging information that we
> need.

Does the application need to know information that the user wants to
know? The operator on the acceptor side wants to know who connected to
him, for example. That isn't necessarily debug information. What about
the javascript security issues that brought about DNS pinning in web
browsers? Do we declare that out of bounds for TCP when multipath is
operational? Or do we say: hey, that's a neat concept. We should
continue support by allowing the app to know what candidate addresses
were negotiated during the original HTTP connection so that the whole
list can be used in subsequent HTTP connections?

> To take a common problem, consider PathMTU discovery.
> Some firewalls block ICMP messages and this breaks
> administrators without requiring any change to application and
> without needing to extend the applications to deal with these
> problems.

I would argue that TCP's PMTUD is one of the most broken things about
TCP's current design. It isn't just badly behaving firewalls... There
are any number of perfectly rational network designs in which there is
a path between two endpoints but not necessarily paths between either
endpoint and all intermediate routers.

The TCP stack sits there like an idiot retransmitting a packet that
never gets through but won't consider the possibility of an MTU
problem unless triggered by the intermediate system.

You should look at the grief the folks on the RRG went through with
the various map-encap designs trying to assure that the intermediate
tunnel encoders could successfully receive and translate a
fragmentation-needed message, all because TCP breaks without it.
Several nasty design constraints at the routing layer would vanish if
TCP could identify the path MTU through packet loss without active
notification of failure.

At any rate, mutlipath TCP needs to be able to quickly determine that
a subflow has failed, regardless of the behavior of intermediate
systems. If it can't, we could end up with a level of reliability
that's actually lower that TCP without multipath.

> need to be exposed to the applications

Keepalives *are* exposed via SO_KEEPALIVE in setsockopt, and I think
that's sensible. My question was: is there any reason to handle the
general use of SO_KEEPALIVE on subflows differently than on the
connection as a whole? I can imagine some instances where TCP might
want to send a one-time keepalive on a subflow because it has reason
to suspect the subflow has failed and wants to decide whether to kill
it, but I'm not sure there's a reason to expose that to the API.

> http://www.stuartcheshire.org/IETF72/

That server is dead at the moment. I'll give it another try later...

Regards,
Bill Herrin

---

From: Narasimhan Venkataramaiah <[email-masked]>
Date: Sat, 14 Nov 2009 02:06:17 +0000

While problems with PMTU discovery are orthogonal to the discussion on the involvement of an application in the multipath tcp operation 

"The TCP stack sits there like an idiot retransmitting a packet that
never gets through but won't consider the possibility of an MTU
problem unless triggered by the intermediate system."

this is fixed in regular TCP with black hole detection logic as described in RFC 2923 and should be applicable to multipath TCP as well.

Simha

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of William Herrin
Sent: Friday, November 13, 2009 5:43 PM
To: [email-masked]
Cc: [email-masked]; SCHARF, Michael
Subject: Re: [multipathtcp] Initial ideas for API requirements

Hi Olivier,

Regards,
Bill Herrin

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 19 Nov 2009 10:16:44 +0100

> How about redundancy? I can think of cases where the app 
> might like to send all segments on all paths, to reduce end 
> to end rtt, etc.

Good point. However, I think that this only makes sense in selected use
cases, given that it can significantly increase the total amount of sent
traffic. I have the impression that one mostly would want to use this
feature for low-bandwidth interactive applications (telnet, voice, ...),
and in particular if the paths are not significantly congested.

One might be able to implicitly detect some of these situations inside
the stack and then send all packets on all paths, but I think that there
should be an (optional, as always) control API as well.

Michael

---

From: Lars Eggert <[email-masked]>
Date: Thu, 19 Nov 2009 11:37:34 +0200

Hi,

agree with Olivier for the most part.

I think we need to distinguish what information should be offered to applications via the socket API, and which information should be offered via a management interface (e.g., a revision to RFC4022/RFC4898). The requirements list below seems to be mixing the two kinds.

Lars

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 19 Nov 2009 13:36:05 +0100

Olivier, 

thanks a lot for the feedback. I just comment here on the application
profile; the other aspects have already been discussed in the thread.

> > However, in MPTCP this does make a _real_ difference. This is not 
> > simply a case of saying "do I want the best QoS or not the 
> best QoS" 
> > (for which there's no wonder an application always says 
> they want the 
> > best). This is a case of "do I want throughput, or reduced 
> jitter" for example.
> 
> i'm not convinced that application developpers will want to 
> deal with these details. We need to keep the API as simple as 
> possible (and perhaps simpler than that)

True, the API must be simple. However, app developers also learnt to use
TCP_NODELAY, which basically distinguishes two profiles (interactive
app. vs. bulk data transport). If it turns out that MPTCP could really
benefit from some third profile, I guess that app developers could get
familiar to this as well. But, actually, we don't know yet whether we
need a distinction beyond TCP_NODELAY - this is all to be discussed.

> For me, profile is similar to traffic contract in intserv, I 
> don't expect that many applications will define their 
> profile. Most applications will use the default profile and 
> this what we should optimise

I disagree with the comparison to IntServ. The application profiles
mentioned in the draft only affect how data is assigned to the different
subflows, which significantly affects the delay seen by the application.
This is quite similar to TCP_NODELAY. Of course, there are similarities
to some DiffServ classes (e. g., RFC 4594), but we are not talking here
about a set of 12 DiffServ classes, but a very small amount of choices
only.

I still think that if the MPTCP stack offers, for instance, two or three
different ways how sub-flows are used, and if their difference is easily
understandable, app developers could make an informed choice that might
be better than an implicit heuristic inside the MPTCP stack. Obviously,
as MPTCP will be optimized for bulk data transfer, most TCP-based apps
will be happy with the default settings and will not use any API
extensions anyway.

Michael

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 19 Nov 2009 13:39:24 +0100

Olivier, Lars, all,

> agree with Olivier for the most part.

first, let me state once again that this is a list of *potential*
requirements, i. e., it is obvious that some of the items will not be
considered finally. It is IMHO very important to discuss which ones do
not make sense - and that is the purpose of this large list.

Still, I do believe that it is reasonable to define a number of
(optional!) API extensions. In addition to what has already been said, I
want to mention two additional aspects:

On the one hand, the interface between the TCP/IP and apps is actually
not simple even today. For instance, the Linux TCP stack has quite a
number of (partly non-interoperable) socket options in addition to
TCP_NODELAY (e. g., TCP_CORK, TCP_DEFER_ACCEPT, TCP_INFO, TCP_KEEPCNT,
TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_LINGER2, TCP_MAXSEG, TCP_QUICKACK,
TCP_SYNCNT, TCP_WINDOW_CLAMP, ...). One can certainly question whether
all of them are indeed required, or whether this is good API style, but
most of these options have probably been implemented because some app
developer at some point in time actually needed exactly this option. And
IMHO we do have to avoid that each MPTCP stack finally provides its own
non-interoperable set of API extensions.

On the other hand, instead of MPTCP, an app can always use multiple
several TCP connections with an own data striping mechanism. In this
case an app has full control. MPTCP to some extent provides an
equivalent transport mechanism inside the TCP/IP stack. Thus, when
discussing the MPTCP API, we must also compare it to the situation of an
app that implementes its own data striping, with a lot of degrees of
freedom. (Many of which will be handled by MPTCP internally and
automatically without app control).

> I think we need to distinguish what information should be 
> offered to applications via the socket API, and which 
> information should be offered via a management interface 
> (e.g., a revision to RFC4022/RFC4898). The requirements list 
> below seems to be mixing the two kinds.

Correct. The sockets interface to the app is indeed not the only
user-space MPTCP interface. In addition, there will of course be a
configuration interface for setting the default configuration, and other
possibilities to retrieve statistics from the stack (e. g., MIB). 

In some sense, the latter ones are out-of-band interfaces, whereas the
socket API is some form of in-band API. The key question is: What
information/configuration must be accessible from an individual app? I
think that an app is more likely to use an in-band API style, instead
of, e. g., a MIB.

Michael

---

From: Zhen Cao <[email-masked]>
Date: Thu, 19 Nov 2009 21:45:52 +0800

Hi All,

Since there is an open source implementation of MPTCP, I think every
body would like to know if the implementors have changed any existing
API or find any existing API inadequate for the implementation. That
will help the document specification a lot.

Thanks,
Zhen
China Mobile