Subject: [multipathtcp] API: which address to use

From: Pasi Sarolahti <[email-masked]>
Date: Mon, 9 Nov 2009 19:49:16 +0900

Hi,

This problem was just discussed in the meeting. Could one possible  
solution to the getsockname semantics problem be to use a separate  
address family for multipath connections? It could be similarly  
structured to the traditional AF, but from the different AF the app  
would know the semantics are different.

Separate AF also explicitly tells that multipath TCP is enabled.

This would not be fully backwards compatible with legacy apps, though,  
which I guess was one of the design goals.

- Pasi

---

From: William Herrin <[email-masked]>
Date: Mon, 9 Nov 2009 08:42:32 -0500

On Mon, Nov 9, 2009 at 5:49 AM, Pasi Sarolahti <[email-masked]> wrote:
> This problem was just discussed in the meeting. Could one possible solution
> to the getsockname semantics problem be to use a separate address family for
> multipath connections? It could be similarly structured to the traditional
> AF, but from the different AF the app would know the semantics are
> different.

Hi Pasi,

Not really, no.

Backwards compatibility is an issue, as you mentioned. So is a desire
to be able to use both IPv4 and IPv6 paths in a single multipath TCP
connection. If it's a legacy app then it's expecting getsockname to
return the same address family that it used to initiate the socket. If
you provide something else you could open a path for buffer overflows
or other mischief.

For applications which are aware of Multipath TCP, the underlying
address is quite literally the wrong question. The connection has
candidate addresses, active subflows and some form of shared
identifier but it doesn't have "a" local address. Getsockname and
getpeername should fail with EMULTIPATH.

Regards,
Bill Herrin

-- 
William D. Herrin ................ [email-masked]  [email-masked]
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: William Herrin <[email-masked]>
Date: Mon, 9 Nov 2009 09:15:46 -0500

>> AF, but from the different AF the app would know the semantics are

I'm going to back up the train on that one.

I stand by what I said about getsockname, however specifying a
PF_INETMP in the original socket call would be more appropriate than
PF_INET or PF_INET6 if we intend multipath to be protocol family
agnostic. It would also be a reasonable way for the app to signal to
the stack whether it expects the use of multipath.

Regards,
Bill Herrin

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 10 Nov 2009 14:01:59 +0900

On Nov 9, 2009, at 10:42 PM, William Herrin wrote:

>> solution

Ok, so there could be a non-backwards compatible way and a  
(pseudo)backwards compatible way. I don't think these had to be  
exclusive, and both could be supported at the same time.

Non-backwards compatible version would be to use new AF_MULTIPATH,  
that would be an array of AF_INET and AF_INET6 entries. Of course that  
would need to be used consistently throughout the connection lifetime.  
gethostname and getpeername would return the current set of active  
addresses at local or remote end. bind() could use this to bind to  
several addresses at a time. connect() could use this to give more  
than one known destination addresses to connect to (although the MPTCP  
protocol logic may prevent the actual connect operation from happening  
simultaneously).  gethostbyname might return several addresses from  
DNS in a single AF_MULTIPATH record, to be directly used by  
connect(AF_MULTIPATH), instead of needing to cycle through the  
addresses separately, as a traditional single-homed version would do.  
This would require modifications also elsewhere than TCP code, but  
that's what you get for having an API that exposes the IP address to  
the application and assumes only one address is in use at a time.

The backwards compatible version could just return the oldest active  
address and hope nothing breaks on legacy apps. I don't know if this  
is better or worse choice than returning the first used address that  
might not exist anymore. Has it been specified anywhere that  
getsockname/getpeername should return same result through the  
connection lifetime?

- Pasi

---

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 12 Nov 2009 03:11:01 +0100

> I stand by what I said about getsockname, however specifying 
> a PF_INETMP in the original socket call would be more 
> appropriate than PF_INET or PF_INET6 if we intend multipath 
> to be protocol family agnostic. It would also be a reasonable 
> way for the app to signal to the stack whether it expects the 
> use of multipath.

I think that a PF_INETMP could indeed be a non-mandatory way to enable
MPTCP (MPTCP would still also be used with PF_INET or PF_INET6 in order
to be backward compatible). Are there any other opinions on the
PF_INETMP solution?

Michael

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 23 Nov 2009 09:22:14 +0900 (JST)

Hello Pasi,

I like the idea. 
But I have several questions. I think there might be an alternative: having 
SOCK_MULTISTREAM or using a combination socket type like SOCK_STREAM | SOCK_MULTIPATH 
from the following reasons.

 Strictly speaking, AF_MULTIPATH is not a new address type. 

 SCTP socket can be created with AF_INET or AF_INET6, but the socket can
 support multiple addresses. It's a bit orthgonal behavior from AF_MULTIPATH

 If the main point of having AF_MULTIPATH is to distinguish between the current 
 TCP socket and multipath TCP socket, using SOCK_MULTISTRAM is not a problem.

 we might support one-ended approach with the same API.
 
Also, in my feeling, we will need new bind(), connect() to specify multiple 
addresses. just like sctp_bindx(), sctp_connectx(). 

Thanks,
--
Yoshifumi Nishida
[email-masked]

From: Pasi Sarolahti <[email-masked]>
Subject: Re: [multipathtcp] API: which address to use
Date: Tue, 10 Nov 2009 14:01:59 +0900
Message-ID: <[email-masked]>

 > would need to be used consistently throughout the connection
 > lifetime. gethostname and getpeername would return the current set of
 > active addresses at local or remote end. bind() could use this to bind
 > to several addresses at a time. connect() could use this to give more
 > addresses separately, as a traditional single-homed version would
 > do. This would require modifications also elsewhere than TCP code, but
 > _______________________________________________
 > multipathtcp mailing list
 > [email-masked]
 > https://www.ietf.org/mailman/listinfo/multipathtcp

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 23 Nov 2009 09:44:04 +0900 (JST)

Hello people,

Sorry for the delay. I've just uploaded the draft version of the minute at IETF76.
Please check the following URL to see if there is any problem.

http://www.ietf.org/proceedings/09nov/minutes/mptcp.txt

Thanks for everyone to attend very active and productive discussions, also
we really appreciate Scott and Andrew for very precise note taking.

Cheers,
--
Phil and Yoshifumi

---

From: Pasi Sarolahti <[email-masked]>
Date: Mon, 23 Nov 2009 13:40:02 -0800

Hi,

(changed the topic line, it was earlier "API: which address to use".  
I'm responding to Yoshifumi below, but first wanted to post a general  
note.)

I wrote down some further thinking about the idea of multipath address  
family in the form of Internet-Draft, which I thought to be better  
than a long Email to the list. The draft is available at http://tools.ietf.org/html/draft-sarolahti-mptcp-af-multipath-00 
  and I'd be happy to hear further comments about it.

I think a new address family would be a minimal needed change to the  
current socket API, without extending the API with new operations, but  
yet provide an interface to handle multiple addresses in a socket. One  
potentially useful characteristic would be that this would allow name  
resolver to return multiple IPv4 and/or IPv6 addresses from the name  
server as a single sockaddr structure, that could be directly given as  
a parameter to a connect call, which would lead to trying which of the  
addresses are reachable for the MPTCP session. Traditionally this has  
been done in the applications, by cycling through the returned  
addresses from resolver one by one, until connect is successful.  
Especially dual-stack IPv6/IPv4 hosts should find such feature useful.

Yoshifumi- thanks for your comments! Some responses inline below.

> from the following reasons.

I have thought the 'type' parameter in the socket call to specify the  
semantics of the socket functions such as read and write. To my  
understanding the purpose of the MPTCP exercise is to maintain similar  
read/write semantics to TCP, so if successful in this goal, the socket  
behavior should be equal to SOCK_STREAM. Or to put it differently,  
MPTCP is not a new protocol, it is just a (fairly large) modification  
to TCP.

> Strictly speaking, AF_MULTIPATH is not a new address type.

I see it as a kind of a super - address family: union of existing  
address families. But I think that would still fit under a definition  
of "address family". I understand that there might be different  
interpretations of what "address family" means, but if there is one  
specified & correct definition, I'd like to learn about it.

> support multiple addresses. It's a bit orthgonal behavior from  

Yes, earlier protocols have done something different for their API.  
But I'd be interested to learn if there is something why the earlier  
protocols couldn't begin using something like multipath address family.

> TCP socket and multipath TCP socket, using SOCK_MULTISTRAM is not a  

In my opinion, the main purpose is to be able to present multiple  
addresses associated with the same TCP socket. But different address  
family could be used to distinguish between multi- and singlehomed  
operations, too.

> we might support one-ended approach with the same API.

Yep, and that should be possible with AF_MULTIPATH, by having just one  
address in the address list. I think there might be cases where even  
single-homed TCP connections might benefit from multiaddress API, even  
if they ended up using just one of the given addresses at both ends  
(referring to the name resolver example above).

> addresses. just like sctp_bindx(), sctp_connectx().

I think that adding completely new operations to the good old socket  
API might be problematic, and would lead to a more updating /  
maintenance burden in different systems and applications than a new  
address family would do. But I might be missing something in the  
required complexity of adding a new address family.

- Pasi

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 24 Nov 2009 11:20:14 +0100

Pasi,
> 
> (changed the topic line, it was earlier "API: which address to use". I'm
> responding to Yoshifumi below, but first wanted to post a general note.)
> 
> family in the form of Internet-Draft, which I thought to be better than
> a long Email to the list. The draft is available at
> http://tools.ietf.org/html/draft-sarolahti-mptcp-af-multipath-00 and I'd
> be happy to hear further comments about it.
> 
> server as a single sockaddr structure, that could be directly given as a
> parameter to a connect call, which would lead to trying which of the
> been done in the applications, by cycling through the returned addresses
> from resolver one by one, until connect is successful. Especially
> dual-stack IPv6/IPv4 hosts should find such feature useful.

This could be interesting, but if we change de socket API, why don't we
define a new structure that can handle DNS names in addition to IPv4 and
IPv6 addresses. This would allow some applications to simply call
connect with a DNS name and let the transport protocol figure out
whether single path IPv4, single path IPv6, multipath whatever ...
should be used to reach this DNS name.

Olivier

-- 
http://inl.info.ucl.ac.be , UCLouvain, Belgium

---

From: William Herrin <[email-masked]>
Date: Tue, 24 Nov 2009 07:34:48 -0500

On Mon, Nov 23, 2009 at 4:40 PM, Pasi Sarolahti <[email-masked]> wrote:
> I think a new address family would be a minimal needed change to the current
> socket API, without extending the API with new operations, but yet provide
> an interface to handle multiple addresses in a socket. One potentially
> useful characteristic would be that this would allow name resolver to return
> multiple IPv4 and/or IPv6 addresses from the name server as a single
> sockaddr structure, that could be directly given as a parameter to a connect
> call, which would lead to trying which of the addresses are reachable for
> the MPTCP session.

Hi Pasi,

It seems like you want to preserve the connect() call's parameters by
altering the definition of sockaddr so that it can contain multiple
addresses.

But functionally, what we really want to do is pass an array of
sockaddrs to connect().

Stupid question time: what makes redefining sockaddr superior to an
extended API call that supplants connect() and accepts an array of
sockaddrs?

Poking around linux's include files (netinet/in.h, bits/socket.h) I
notice that they go to a lot of trouble to make all sockaddrs come out
to exactly the same number of bytes. Won't a variable length sockaddr
that holds a variable number of addresses make a mess of that?

>> from the following reasons.

Found a quote I think bears on the topic at
http://lkml.indiana.edu/hypermail/linux/net/9904.2/0091.html :

>The AF_ prefix stands for "address family" and the PF_prefix stands for
>"protocol family". Historicly, the intent was that a single protocol family
>might support multiple address families and that the PF_ value was used to
>create the socket and the AF_ value was used in socket address structures.
>But in actuality, a protocol family supporting multiple address families has
>never been supported and the <sys/socket.h> header defines the PF_ value
>for a given protocol to be equal to the AF_ value for that protocol.

I think when we discuss allowing multipath TCP to use IPv4 and IPv6
addresses interchangeably, we're talking about one protocol family
using multiple address families, are we not?

Regards,
Bill Herrin

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 24 Nov 2009 22:04:53 +0100

On Nov 23, 2009, at 10:40 PM, Pasi Sarolahti wrote:

> Hi,
> 
> (changed the topic line, it was earlier "API: which address to use". I'm responding to Yoshifumi below, but first wanted to post a general note.)
> 
> I wrote down some further thinking about the idea of multipath address family in the form of Internet-Draft, which I thought to be better than a long Email to the list. The draft is available at http://tools.ietf.org/html/draft-sarolahti-mptcp-af-multipath-00 and I'd be happy to hear further comments about it.
> 
> I think a new address family would be a minimal needed change to the current socket API, without extending the API with new operations, but yet provide an interface to handle multiple addresses in a socket. One potentially useful characteristic would be that this would allow name resolver to return multiple IPv4 and/or IPv6 addresses from the name server as a single sockaddr structure, that could be directly given as a parameter to a connect call, which would lead to trying which of the addresses are reachable for the MPTCP session. Traditionally this has been done in the applications, by cycling through the returned addresses from resolver one by one, until connect is successful. Especially dual-stack IPv6/IPv4 hosts should find such feature useful.
There is one thing that I do not understand. Currently there is no way a DNS
server resolves a name indicating that all addresses belong to the same host.
You just get a list of addresses. They might belong to one host or not.

At least for SCTP this is important and makes it impossible to use DNS to
resolve a name to multiple addresses belonging to the same host.

Best regards
Michael
> I have thought the 'type' parameter in the socket call to specify the semantics of the socket functions such as read and write. To my understanding the purpose of the MPTCP exercise is to maintain similar read/write semantics to TCP, so if successful in this goal, the socket behavior should be equal to SOCK_STREAM. Or to put it differently, MPTCP is not a new protocol, it is just a (fairly large) modification to TCP.
> 
> I see it as a kind of a super - address family: union of existing address families. But I think that would still fit under a definition of "address family". I understand that there might be different interpretations of what "address family" means, but if there is one specified & correct definition, I'd like to learn about it.
> 
> Yes, earlier protocols have done something different for their API. But I'd be interested to learn if there is something why the earlier protocols couldn't begin using something like multipath address family.
> 
> In my opinion, the main purpose is to be able to present multiple addresses associated with the same TCP socket. But different address family could be used to distinguish between multi- and singlehomed operations, too.
> 
> Yep, and that should be possible with AF_MULTIPATH, by having just one address in the address list. I think there might be cases where even single-homed TCP connections might benefit from multiaddress API, even if they ended up using just one of the given addresses at both ends (referring to the name resolver example above).
> 
> I think that adding completely new operations to the good old socket API might be problematic, and would lead to a more updating / maintenance burden in different systems and applications than a new address family would do. But I might be missing something in the required complexity of adding a new address family.
> 
> - Pasi
>

---

From: Michael Tüxen <[email-masked]>
Date: Tue, 24 Nov 2009 22:07:36 +0100

Dear all,

just one note:
In SCTP we use sctp_connectx() to connect to multiple addresses.
Kacheong pointed a couple of years ago out that a more generic
connectx() would make sense to other protocols supporting multihoming
as well (but there were no).

So you might want to look at sctp_connectx() which the SCTP socket API
uses to handle the problem.

Best regards
Michael

---

From: "Laganier, Julien" <[email-masked]>
Date: Tue, 24 Nov 2009 14:32:58 -0800

Stuart Cheshire had a nice talk at IETF 72 on the drawbacks of the current getaddrinfo() + connect() model w.r.t. to IPv6 transition,see http://www.stuartcheshire.org/IETF72/. (Somebody already mentioned that on the list I think.)

A connect_to_name() calls that hides all of the address family subtleties under the hood
does not have such drawbacks. So if we're going to change the API anyway, this looks like a good occasion to fix that issue too, no?

--julien 

> -----Original Message-----
> From: [email-masked] [mailto:multipathtcp-
> [email-masked]] On Behalf Of Michael Tüxen
> Sent: Tuesday, November 24, 2009 1:08 PM
> To: William Herrin
> Cc: [email-masked]
> Subject: Re: [multipathtcp] Multipath Address Family
> 
> Dear all,
> 
> just one note:
> current
> >>> SOCK_MULTISTREAM or using a combination socket type like
> SOCK_STREAM |
> >>> SOCK_MULTIPATH

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Nov 2009 14:38:50 -0800

> does not have such drawbacks. So if we're going to change the API anyway, this looks like a good occasion to fix that issue too, no?

I like this idea, but there needs to be a returned primary IP address if
this is intended to support TCP. I.e., you can ask the system to call a
name, but it ends up establishing a TCP connection to a socket pair (src
IP, dst IP, src port, dst port).

I'm still not sure what that means for multipath. There could be
secondary socket pairs of course, and they can vary over the life of the
connection, but I still feel that if the primary disappears the
connection MUST disappear *IF* this is TCP.

Of course, if it's not TCP, this could be easily defined in other more
useful ways.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Tue, 24 Nov 2009 14:56:38 -0800

Hi, Joe,
> 
> Laganier, Julien wrote:
> > Stuart Cheshire had a nice talk at IETF 72 on the drawbacks of the
> > current getaddrinfo() + connect() model w.r.t. to IPv6 transition,see
> > http://www.stuartcheshire.org/IETF72/. (Somebody already mentioned that
> > on the list I think.)
> >
> > does not have such drawbacks. So if we're going to change the API
> > anyway, this looks like a good occasion to fix that issue too, no?
> 
> I like this idea, but there needs to be a returned primary IP address
> if this is intended to support TCP. I.e., you can ask the system to call
> a name, but it ends up establishing a TCP connection to a socket pair
> (src IP, dst IP, src port, dst port).
> 
> secondary socket pairs of course, and they can vary over the life of
> the connection, but I still feel that if the primary disappears the

Without taking position with respect to fate-sharing between the MPTCP connection and the original address-port pair, whether the MPTCP connection is torn down when the original address-port pair fails is, I think, an issue orthogonal to the specifics of how an application connects to a peer.

In both case we can introduce the connect_to_name() call in the new API.

Orthogonal to that, if we decide we want to treat the initial address pair() that got used to establish the MPTCP connection in a special fashion, e.g., fate-sharing, we can still do so by defining the behavior in the protocol specification.

Makes sense?

--julien

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Nov 2009 15:00:32 -0800

Laganier, Julien wrote:
> Without taking position with respect to fate-sharing between the MPTCP
> connection and the original address-port pair, whether the MPTCP
> connection is torn down when the original address-port pair fails is, I
> think, an issue orthogonal to the specifics of how an application
> connects to a peer.
> 
> Orthogonal to that, if we decide we want to treat the initial address 
> pair() that got used to establish the MPTCP connection in a special
> fashion, e.g., fate-sharing, we can still do so by defining the behavior
> in the protocol specification.
> 
> Makes sense?

Sure, but either way you need to return at least the primary IP
address/port pair as results of the connect_to_name() call.

It's not sufficient to return an undistinguished set of such pairs, nor
is it sufficient to not return any such pairs.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Tue, 24 Nov 2009 15:08:53 -0800

Joe Touch wrote:
> >>> current getaddrinfo() + connect() model w.r.t. to IPv6 transition,
> >>> see http://www.stuartcheshire.org/IETF72/. (Somebody already mentioned
> >>> that on the list I think.)
> >>>
> >> I like this idea, but there needs to be a returned primary IP
> >> address if this is intended to support TCP. I.e., you can ask the system
> >> to call a name, but it ends up establishing a TCP connection to a socket
> >> pair (src IP, dst IP, src port, dst port).
> >>
> >> Of course, if it's not TCP, this could be easily defined in other
> >> more useful ways.
> >
> > Without taking position with respect to fate-sharing between the
> > MPTCP connection and the original address-port pair, whether the MPTCP
> > connection is torn down when the original address-port pair fails is,
> > I think, an issue orthogonal to the specifics of how an application
> > connects to a peer.
> >
> > fashion, e.g., fate-sharing, we can still do so by defining the
> > behavior in the protocol specification.
> >
> > Makes sense?
> 

Unless the socket has been bind()'ed earlier, connect() doesn't return to the caller any information about the address/port that was picked as a result of the connect(). The application would need to call getsockname() to know what address/port pair was used. 

If an application is modified to use the new API, it will have to use getpeername() in addition to getsockname() to know what initial address pair has been connected.

--julien

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Nov 2009 15:28:19 -0800

Laganier, Julien wrote:
> Unless the socket has been bind()'ed earlier, connect() doesn't
> return
> to the caller any information about the address/port that was picked as
> a result of the connect(). The application would need to call
> getsockname() to know what address/port pair was used.

Sure. I was speaking in a general sense.

> If an application is modified to use the new API, it will have to use
> getpeername() in addition to getsockname() to know what initial address
> pair has been connected.

That basically kills legacy apps. You want getsockname to return the
initial, and some other call to return the full list (or supplemental
socket pairs).

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Tue, 24 Nov 2009 15:34:45 -0800

Joe Touch wrote:
> >> [...]
> >> It's not sufficient to return an undistinguished set of such pairs,
> >> nor is it sufficient to not return any such pairs.
> >
> > return to the caller any information about the address/port that
> > was picked as a result of the connect(). The application would need
> > to call getsockname() to know what address/port pair was used.
> 
> > getpeername() in addition to getsockname() to know what initial
> > address pair has been connected.
> 

Hmm... Not sure I understand your statement about legacy apps. 

We've been talking about an app using a new connect_to_name() API call, so it's no longer entirely legacy -- If it wants to use the new connect_to_name() API call, it has to use getpeername() to get its peer address, that's it. 

I do not see how that is related to legacy apps, nor while it would be killing them :)

--julien

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 24 Nov 2009 15:37:53 -0800

> should be used to reach this DNS name.

I agree that connecting by names would eventually be the right way to  
go for Internet protocols, but am not sure if that fits within the  
authority of this WG anymore. It may be something that needs to be  
discussed by a wider community. My proposal was less ambitious and  
more short-sighted: fix the shortcoming in the API that claims the  
socket is using only one address when it is in fact using many.

Another question is that even if we had an connect-by-name API, would  
we still need to be able express which address(es) are bound to the  
socket in the API?

- Pasi

---

From: Joe Touch <[email-masked]>
Date: Tue, 24 Nov 2009 15:43:58 -0800

> I do not see how that is related to legacy apps, nor while it would be killing them :)

Agreed. But I can port a legacy app easier if these interfaces augment,
but don't change, the semantics.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 25 Nov 2009 09:05:19 +0900 (JST)

Hello Pasi,

Thanks for your response. 

From: Pasi Sarolahti <[email-masked]>
Subject: Multipath Address Family
Date: Mon, 23 Nov 2009 13:40:02 -0800
Message-ID: <[email-masked]>

 > MPTCP is not a new protocol, it is just a (fairly large) modification

I would say that the socket type indicates the semantics of communication. 
(as 'man socket' says.)  If communication style has been changed, we might need 
new one. If the transport service is completely the same from the API's point 
of view even if multipath is used or not, I think you're right. 
But, if you want to add extentions to streaming service, it can be a different 
communication semantic.

 > Yes, earlier protocols have done something different for their
 > API. But I'd be interested to learn if there is something why the
 > earlier protocols couldn't begin using something like multipath
 > address family.

Yep, if SCTP supports this kind of AF, things might be easier..
Actually, in WIDE project, we tried to propose new AF called AF_BUNDLE for sctp, 
which has very similar concept of AF_MULTIPATH. An experimental code existed, 
but somehow it was not integrated into KAME's code... 

Anyway, we already have a standardized multipath stream protocol. 
it would not be very perferable if the API for two multipath stream 
protocols are very different. 
I'm not sure how this is important yet, but that's one point I care.

 > if they ended up using just one of the given addresses at both ends

Yes. it's possible. but, I'm not very sure if it's intuitive...

 > required complexity of adding a new address family.

Hmm, in your proposal, it seems to me that you want to update the 
return value for bind() and connect(), which might break the syntax of these calls.
If we want to keep the same syntax as the current one, we will need to have a 
new function that will get true return code for these functions. 
But, in my feeling, this might be almost the same as creating new bind() and connect().

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: "Laganier, Julien" <[email-masked]>
Date: Tue, 24 Nov 2009 16:13:03 -0800

Joe Touch wrote: 
> >>> If an application is modified to use the new API, it will have to
> >>> use getpeername() in addition to getsockname() to know what initial
> >> initial, and some other call to return the full list (or
> >> supplemental socket pairs).
> >
> > We've been talking about an app using a new connect_to_name() API
> > call, so it's no longer entirely legacy -- If it wants to use the new
> > connect_to_name() API call, it has to use getpeername() to get its peer
> > address, that's it.
> >

As I said earlier, the fate sharing semantics between the original address/port pair and the connection is IMHO a protocol matter. Whatever decision we take there, we can implement it in the existing and new APIs.

--julien

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 25 Nov 2009 09:14:14 +0900 (JST)

From: Michael Tüxen <[email-masked]>
Subject: Re: [multipathtcp] Multipath Address Family
Date: Tue, 24 Nov 2009 22:07:36 +0100
Message-ID: <[email-masked]>

 > as well (but there were no).

Just curiosity, do you know the reason why generic connectx() doesn't exist? 
it make sense to me since the call looks independent from sctp.
Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 24 Nov 2009 17:52:06 -0800

> extended API call that supplants connect() and accepts an array of

Another question: is it better to have a compile-time error (or  
segmentation fault) or run-time error code, if application was built  
with multi-address support, but system was not?

Even if we had a new mptcp_connectx, bindx, etc. calls, the problem  
still remains what to do with the old socket operations that assume  
only single address per endpoint. A new address family would be one  
(unfortunately non-backwards compatible) way to address that problem  
(possibly with the cost of introducing new problems).

I think there is a certain elegancy in having a minimal socket API  
with just a small basic set of operations. Until now the same socket  
API has provided to be flexible enough to support many uses over time  
(Unix domain sockets that take pathname as sockaddr, IPsec PF_KEY  
sockets, netlink sockets, ...). I'm not yet ready to buy it that we  
have come to a point that the old API is not sufficient anymore, if  
the same functionality could be implemented by specifying a couple of  
new socket parameters for a feature that is supported by just some of  
the protocols under the API.

> that holds a variable number of addresses make a mess of that?

I guess the question is whether the length of sockaddr is a Linux  
implementation issue that can be fixed, or whether there is some  
deeper reason for having the sockaddrs to be of same length. In Unix  
domain sockets the pathnames in sockaddr can be variable length and  
quite long, to my understanding, certainly worth of several IP  
addresses. But it would be interesting to know what, if anything,  
would break in legacy systems with a new sockaddr variant. I'm  
assuming that the TCP stack needs to go through fairly significant  
change anyway because of MPTCP.

> using multiple address families, are we not?

I have been curious about this PF/AF dependence that seems like a long- 
standing "bug" in some systems. Have you ever investigated this  
deeper? Why would it be so hard to fix it so that you could have  
several address families in one protocol family?

- Pasi

---

From: Andrew Yourtchenko <[email-masked]>
Date: Wed, 25 Nov 2009 03:12:35 +0100 (CET)

On Tue, 24 Nov 2009, Joe Touch wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> Laganier, Julien wrote:
>> Joe Touch wrote:
>>> Laganier, Julien wrote:
>>>> Joe Touch wrote:
>>>>> [...]

FWIW (apologies to get in the middle of the conversation) - I had 
looked at a couple of browser apps when working on the prototype code for 
the "happy eyeballs" draft - my code was implementing the connect_by_name() 
semantics, with two threads racing in the background for IPv4 and IPv6 
host lookup and connect attempts.

I saw the biggest difficulty with porting the apps to use the 
connect_to_name() API in the work required to refactor the code that 
deals with the blocking resolver in a non-blocking manner (along with 
giving the user the visual clues of "Resolving host for 
hostname.example.com..."), selects in some application-specific way the 
sockaddr to use and passes it to the non-blocking socket code, and deals 
with the various recoverable error conditions (like retrying with other 
address).

But, that said, I think with the right hooks for very specific cases where 
the application needs the fine-grained control over what exactly the 
"remote hostname" should boil down to, connect_by_name() could save 
quite a lot of "boilerplate coding" for the future.

thanks,
andrew

---

From: Pasi Sarolahti <[email-masked]>
Date: Tue, 24 Nov 2009 18:51:04 -0800

> with multi-address support, but system was not?

Uh.. forget what I said about compile-time errors... But I would think  
not changing the API would allow for more elegant error handling when  
multi-address apps are run on old systems that do not support the  
feature.

- Pasi

---

From: Andrew Yourtchenko <[email-masked]>
Date: Wed, 25 Nov 2009 04:31:47 +0100 (CET)

On Wed, 25 Nov 2009, Andrew Yourtchenko wrote:

>>> We've been talking about an app using a new connect_to_name() API call, so 
>>> it's no longer entirely legacy -- If it wants to use the new 
> FWIW (apologies to get in the middle of the conversation) - I had looked at a 
> couple of browser apps when working on the prototype code for the "happy 
> eyeballs" draft - my code was implementing the connect_by_name() semantics, 
> with two threads racing in the background for IPv4 and IPv6 host lookup and 
> connect attempts.

And to correct/clarify myself: I was working with the 
connect_by_name(hostname,servicename) - similar to getaddrinfo()

> connect_to_name() API in the work required to refactor the code that deals 
> with the blocking resolver in a non-blocking manner (along with giving the 
> user the visual clues of "Resolving host for hostname.example.com..."), 
> selects in some application-specific way the sockaddr to use and passes it to 
> the non-blocking socket code, and deals with the various recoverable error 
> conditions (like retrying with other address).
>
> the application needs the fine-grained control over what exactly the "remote 
> hostname" should boil down to, connect_by_name() could save quite a lot of 
> "boilerplate coding" for the future.

thanks,
andrew

---

From: Costin Raiciu <[email-masked]>
Date: Wed, 25 Nov 2009 10:39:00 +0000

Hi,

I have been thinking about this for a bit now, and have chatted with various people from different OSes. 
Murari (from Microsoft) said that they will probably not turn multipath on by default at first release; apps wll be able to use it, but will need to opt in. This will give app writers a chance to update their apps for multipath - since the apps that break are likely a small subset, it seems it will be possible to upgrade these given the time window between OS releases. I would guess most other OSes would do the same in the initial release. So, by default, client hosts will not have multipath "on".

Let's consider mobiles. This is where multipath is badly needed for supporting connection handover between interfaces. Here most of the apps typically run as high level managed code - mostly for security reasons; I bet one can change the stack without breaking these apps even if multipath is on by default. The low level apps that would break are already written mostly by the company running the OS - hence will be natural to upgrade. 
In short, I think multipath could be turned on by default for mobiles, if the mobile providers want this.

Finally, servers: Linux currently has ecn turned on by default for servers, and off on clients. I would assume the same could hold for multipath. This would ensure that initially mobiles can talk multipath to servers, but standard client-server communication is unipath. 

This deployment model seems natural to me, as it gives the much needed benefits to mobiles immediately, and allows the more mature desktop market to slowly sink in multipath.

Coming back to fate sharing of multipath connection and initial subflow: I would assume fate sharing will be a sysctl flag on the host, telling the stack whether the connection should go down or not with the first subflow. I imagine three settings here:
a) connection doesn't go down with first subflow
b) connection dies when local interface dies
c) connection dies when subflow times out (this implies b)

As long as the protocol itself does not make assumptions about fate sharing, the endpoints can do whatever local policy says.  I would imagine mobiles will do policy a), servers can probably do policy b), and clients c) - assuming it is hardest to tell what apps break on client hosts (the apps that use TCP to test path properties will break so they would like c).

I think the combination of these two choices (multipath on/off + fate sharing policy) allows everyone to get what they want, when they want.

Thoughts?
Costin

---

From: Joe Touch <[email-masked]>
Date: Wed, 25 Nov 2009 07:52:53 -0800

Costin Raiciu wrote:
...
> Coming back to fate sharing of multipath connection and initial
> subflow: I would assume fate sharing will be a sysctl flag on the host,
> telling the stack whether the connection should go down or not with the
> first subflow. I imagine three settings here:
>

These are interesting configuration parameters from an implementation
view, but from a protocol view, if we're talking about TCP, there are
only two cases:

	a) connection goes down when the whole set of subflows goes down

	b) connection goes down when the first subflow goes down

What makes a subflow go down shouldn't be defined separately; it's
exactly whatever would make that connection go down if it were a single,
stand-alone TCP connection.

The remaining question will be whether (a) is still TCP, or more
specifically how to make (b) look more like (a).

One way is to say "first subflow packets are tunneled over other
subflows". If that is actually how things are done, then you can claim
that the original subflow is actually still up, and make progress.

However, anything that would cause the original subflow to fail even
when tunneled over other interfaces still, IMO, needs to cause the whole
set to go down. The key example is loss of the IP address associated
with the original flow - either when the interface goes down, or when
it's lost via failed DHCP renewals, or when it's otherwise explicitly
reconfigured.

Provided this latter case is defined clearly enough, this might be a way
to proceed. However, if you start a connection with an address you're
given by an ISP, you can't continue that after the address is revoked.
If you do, you could interfere with other connections from other hosts
to which that address is (re)assigned.

Joe

---

From: "Laganier, Julien" <[email-masked]>
Date: Wed, 25 Nov 2009 08:31:26 -0800

> with multi-address support, but system was not?

Hmm. Neither =)

I think if we want to bind or connect an MPTCP connection to multiple addresses, we want to have new systems calls, in a fashion similar to what was done with sctp_connectx().
 
> (possibly with the cost of introducing new problems).

The "old" socket can have its semantic preserved by exposing through it only the initial address-port pair in use. 

   - connect() will connect to one address-port
   - bind() will bind to one address:port
   - getsockname() will return the address-port to which the socket was first bound
   - getpeername() will return the address-port from which the connection with the peer was initially established. 
 
> the protocols under the API.

AFAICS all uses of the socket API so far had the property that a socket is bound to a _single_ and _constant_ name (e.g., address, pathname), or in other words, there's no more than one name, and the name doesn't change. 

This property is no longer true with MPTCP because a socket can be bound to multiple and changing addresses. So if we're to design an API that exposes that, extending an API that assumed exactly the contrary is IMHO the wrong way to go.

--julien

---

From: Michael Tüxen <[email-masked]>
Date: Wed, 25 Nov 2009 19:20:56 +0100

> it make sense to me since the call looks independent from sctp.
Because we were not sure what the right semantic would be:
1. For SCTP (or other protocols supporting multihoming), we wanted something
   like "establish a transport connection with the peer having *all* given addresses.
2. For TCP (or other protocols not supporting multihoming), we though something
   like "establish a transport connection with a peer having *one* of the given addresses.
This is very different. We also assume that if the user provides multiple address,
all the addresses belong the the *same* host. This makes it impossible to use
DNS for getting them...

So we choose to use sctp_connectx() if if lateron a generic connectx() is defined,
we can easily provide an SCTP semantic, if possible.

Best regards
Michael

---

From: Michael Tüxen <[email-masked]>
Date: Wed, 25 Nov 2009 19:26:11 +0100

> does not have such drawbacks. So if we're going to change the API anyway, this looks like a good occasion to fix that issue too, no?
I'm interested in the userland/kernel boundary, the system calls, not any kind of library calls.
So are you suggesting to do the DNS name lookups in the kernel? This would mean that
you want to implement DLS in the kernel. I'm not sure if I really like that...
We had^H^H^H have a feature in SCTP which also requires DNS lookups right from the
kernel and no-one implemented it. 

I think Stuart was referring to some stuff in CoreFoundation, which are library calls, I think.

Best regards
Michael

---

From: "Laganier, Julien" <[email-masked]>
Date: Wed, 25 Nov 2009 10:38:14 -0800

Michael Tüxen wrote:
> > A connect_to_name() calls that hides all of the address family
> > subtleties under the hood does not have such drawbacks. So if we're
> > going to change the API anyway, this looks like a good occasion to 
> > fix that issue too, no?
>

Good for you.

I'm not talking about any kind of library call, I'm talking about an API, which is what I think the WG is chartered to work on. 

> So are you suggesting to do the DNS name lookups in the kernel? This
> would mean that you want to implement DLS in the kernel. I'm not sure if I really like
> that...

I'm suggesting to do DNS lookups as a result of the connect_to_name() API call, nothing more.

--julien

> >>>> sockaddr structure, that could be directly given as a parameter to
> a
> >> connect
> >>>> The AF_ prefix stands for "address family" and the PF_prefix
> stands
> >> for
> >>>> "protocol family". Historicly, the intent was that a single
> protocol
> >> family

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 26 Nov 2009 04:15:14 +0900 (JST)

Hello Michael, 

From: Michael Tüxen <[email-masked]>
Subject: Re: [multipathtcp] Multipath Address Family
Date: Wed, 25 Nov 2009 19:20:56 +0100
Message-ID: <[email-masked]>

 > all the addresses belong the the *same* host. This makes it impossible to use

Thanks for the clarification. But, I still have a question... Do we have to
connect all given addresses?
If connectx() sends multiple SYNs simultaneously to the multiple destinations,
it will be problematic. But, if connectx() has the following logic,
  it tries to send a SYN one-by-one to each destination
  it gets SYN-ACK, it stops to send a SYN and return.
I don't see a big issue in the logic. It may be slow in some cases, though. 
I mean, connectx() is used only to find one primary destination. Other destinations
will be found during negotiations in the transport protocol. Not from connectx().
--
Yoshifumi Nishida
[email-masked]

---

From: Pasi Sarolahti <[email-masked]>
Date: Wed, 25 Nov 2009 22:55:34 -0800

On Nov 25, 2009, at 8:31 AM, Laganier, Julien wrote:

> The "old" socket can have its semantic preserved by exposing through  
> it only the initial address-port pair in use.
>
>   - getpeername() will return the address-port from which the  
> connection with the peer was initially established.

Ok, so the confusion is about whether get*name really refers to a  
stable "name" that misleadingly looks very much like an address/port,  
or whether it should point to a current address. Posix spec says about  
getpeername that "The getpeername() function shall retrieve the peer  
address of the specified socket", which hints towards the latter  
interpretation, but there may be other references (such as man pages)  
that spell this differently. Man page for getsockname says  
"Getsockname() returns the current address for the specified socket",  
but getpeername talks about names.

If the connection was to survive over the loss of the first address,  
getsockname or getpeername could return an address that has not been  
used by that host for a while, and the returned address might as well  
be already allocated to a different host. This may be a problem for  
applications that have assumed the "current address" interpretation. I  
agree that there needs to be a mode that is backwards compatible with  
the old AF_INET sockets for legacy apps, but if new applications were  
implemented to be multipath-aware with new or extended API, then also  
these calls should return reasonable values that correspond with  
reality.

A different question is how many applications really need to be aware  
of the underlying addresses -- probably a small minority (with  
apologies to people who are fed up with this discussion thread by now).

> AFAICS all uses of the socket API so far had the property that a  
> socket is bound to a _single_ and _constant_ name (e.g., address,  
> pathname), or in other words, there's no more than one name, and the  
> name doesn't change.
>
> This property is no longer true with MPTCP because a socket can be  
> bound to multiple and changing addresses. So if we're to design an  
> API that exposes that, extending an API that assumed exactly the  
> contrary is IMHO the wrong way to go.

I agree that this is different from earlier uses, but why is this a  
problem, if both the application and the protocol implementation have  
a common understanding (indicated by the address family) how the new  
form of sockaddress works? The socket API itself should be transparent  
about the structure of sockaddr, right? Connected socket has a  
property that it has a local and a remote endpoint, but now we have a  
case that an endpoint may have multiple "aliases" to represent it.

- Pasi

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 27 Nov 2009 17:58:27 +0100

> I'm not talking about any kind of library call, I'm talking about an API, which is what I think the WG is chartered to work on. 
Sorry, I was not clear enough. What I meant was that it is important where the
API you are thinking about should be implemented.
If it is at the kernel/userland boundary, like the socket calls, there might be
other restrictions as when you are considering library calls.

The point I wanted to make is that doing DNS lookups in a userland library is simple,
doing it in the kernel is a different story. That is why I wanted to figure out
where you think you API is located at.
> I'm suggesting to do DNS lookups as a result of the connect_to_name() API call, nothing more.
But my point is if it will be in the kernel (if you think your connect_to_name() call
is a system call) or if it is in a library. If it is in the library you still must be able
to implement it using existing system calls.

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 27 Nov 2009 18:06:51 +0100

> connect all given addresses?
You do not have to do anything right now, since it is not defined yet.
The situation for us was the following:
We know the peer has addr1, ... , addrn and we want to establish an SCTP association.
We do not want to test addr1 first, wait until the setup fails and then test addr2,
and so on. We want to setup an association in a fast way not being delayed because
addr1 currently fails. We have not specified if you send INITs to all addresses
simultaneously or send the first INIT to addr1 and the retransmission to addr2, or
in an other way. That is implementation specific. But do no want to wait for the
association setup to addr1 to fail (after several retransmissions) before trying
alternate addresses.
We can not do this with the existing system calls and introduced therefore the
sctp_connectx() system call. 
>  it gets SYN-ACK, it stops to send a SYN and return.
But then it can take long until you setup you transport connection.
The application could just run a loop over the peer addresses and call connect()
to it, see if it works. If it fails try the next. This might take a long time
you you have a lot of addresses to try and some of them are not working.

---

From: Christian Vogt <[email-masked]>
Date: Sat, 28 Nov 2009 03:16:53 -0500

On Nov 24, 2009, Pasi Sarolahti wrote:

> discussed by a wider community.

After my presentation [1] of name-based sockets at the RRG meeting in
Hiroshima, several RRG participants suggested such a wider discussion. 
The idea was to initiate the discussion with a presentation on name-
based sockets at the Applications area, Transport area, or Internet area
open meeting in March.  Would folks in this working group support such a
presentation?

- Christian

[1] http://www.ietf.org/proceedings/09nov/slides/RRG-2.pdf

---

From: Christian Vogt <[email-masked]>
Date: Sat, 28 Nov 2009 03:36:29 -0500

On Nov 24, 2009, Pasi Sarolahti wrote:

> Another question is that even if we had an connect-by-name API, would we
> still need to be able express which address(es) are bound to the socket
> in the API?

Yes.  Whereas the average application using name-based sockets would not
need any information about IP addresses, special applications may.  Hence
even a name-based sockets interface will have to provide hooks through 
which IP address information can be obtained.

- Christian

---

From: Michael Tüxen <[email-masked]>
Date: Mon, 30 Nov 2009 18:28:54 +0100

On Nov 30, 2009, at 4:59 PM, Bryan Ford wrote:

> On Nov 24, 2009, at 4:04 PM, Michael Tüxen wrote:
> You may not know whether all those addresses belong to the same host, but that doesn't necessarily mean they're not useful.  Just for other security reasons, MPTCP will have to have a fairly strong mechanism to verify that whoever is at a given alternate destination address(/port) really is the same host we hold the logical connection with, before starting to use a new subflow to that alternate address(/port).  This is especially true in today's heavily NATted world, where a huge number of endpoints all through the network have highly non-unique IP addresses like '10.1.1.x'; if one MPTCP endpoint says it has an alternate address 10.1.1.5 but is on a different private network from the other endpoint, it's VERY likely that the remote host's attempt to connect to 10.1.1.5 actually WILL reach some host, but the wrong host - even without any malicious attackers anywhere.
> 
> Thus, if we are given a bundle of DNS-resolved addresses, successfully open an initial MPTCP connection to one of them, and then want to try using those other addresses as alternate subflow targets, I don't see an inherent problem with just doing that as long as MPTCP's verification is working right, which it has to anyway.  If the bundle contains addresses of other hosts, those hosts will fail the verification check and subflow creation to them will fail - a few network and host resources get burned, but that can be limited with reasonable policies on attempting new subflows.
So when you connect to (IP1, ..., IPn) you want to connect to one of them. That is fine.
So if your server has n addresses but has no listening scoket, you will exchange n SYS/RST
pairs. (In SCTP we would use only one INIT/ABORT exchange and terminate the association).
> 
> A separate issue is HOW the application "names" a bundle of addresses to the OS.  One option is to add an AF_MULTI or somesuch that takes a bundle of already-resolved addresses, and have the application resolve the DNS name itself to a bundle of addresses and then pass the bundle to the sockets layer using the new AF.  Another option is just to add a new connect_by_name API, or alternately a new AF_NAME address family that embeds a name in its sockaddr struct, and transfer the job of name resolution into the sockets layer.  The latter seems like the cleaner approach to me: as Stuart Cheshire and Christian Vogt and others have pointed out, connect_by_name functions are already getting added to a lot of higher-level networking APIs and thus used by applications: this seems to be the way the industry is going, so all we really need to do is add a way to preserve the naming-based abstraction all the way down to the transport interface (i.e., through the Unix sockets API).
> 
> Cheers,
> Bryan
> 
>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 01 Dec 2009 04:56:29 +0900 (JST)

From: Christian Vogt <[email-masked]>
Subject: Re: [multipathtcp] Multipath Address Family
Date: Sat, 28 Nov 2009 03:16:53 -0500
Message-ID: <[email-masked]>

 > [1] http://www.ietf.org/proceedings/09nov/slides/RRG-2.pdf

With this API, is it possible for applications to specify something like 
"stream service" instead of specifying trasnport protocol? 
I mean, doesn't application have to have fallback codes for multiple transport
protocols by using this API?

I would like to know if we might be able to utilize this idea for gradual
transitions of new trasnport protocols.
--
Yoshifumi Nishida
[email-masked]

---

From: Christian Vogt <[email-masked]>
Date: Mon, 30 Nov 2009 19:13:30 -0500

On Nov 30, 2009, Yoshifumi Nishida wrote:

> With this API, is it possible for applications to specify something
> like  "stream service" instead of specifying trasnport protocol? 

Indeed, Yoshifumi.  And there is flexibility regarding which such
"transport services" applications should be able to explicitly select.
The current version of the paper defines only one service:  stream- 
orientation as opposed to datagram-orientation.  This could be extended
to enable the explicit selection of additional services, such as
reliability or even congestion control.  The current limitation to one
service is a pragmatic choice, which is based on today's small choice of
transport protocols.  But I agree that an ability to define more
services could foster the deployment of new transport protocols.

- Christian