Subject: [Tsvwg] Questions regarding the SCTP Sockets API

From: "Daisy Chang" <[email-masked]>
Date: Thu, 6 Sep 2001 14:15:53 -0400

Hi,
     I was looking into the implementation of the sctp_bindx() call defined
in <draft-ietf-tsvwg-sctpsocket-01.txt>, and I ran into the following
confusions.

     #1. It seems that sctp_bindx() is a new socket interface which can be
used on top of a SCTP socket to add/remove bind addresses at any time.
Therefore, I would like to post this scenario to see if this is a valid
case for using bindx().
Suppose that here are two interfaces on the system, addr1 and addrr2. I
have sk1 bound and listening to addr1.  After I accepted an incoming
association, sk2 is created for the association. I then use bindx() to add
addr2 but remove addr1 on sk1, so later on, new association for sk3 will be
running on addr2. The intention here is to conduct some load balancing
between with sk2 and sk3. This scenario is better illustrated in the
following tables created by La Monte Yaroll.

addr1 = (a1, p0)
addr2 = (a2, p0)

Table 1
=======

System Call         Socket    Local Endpoint  Remote Endpoint     Assoc
-----------         ------    --------------  --------------- -----
sk1 = socket(STREAM)     sk1  NONE       NONE           NONE
bind(sk1, addr1)    sk1       (a1: p0)   NONE           NONE
listen(sk1)         sk1       (a1: p0)   NONE           NONE
sk2 = accept(sk1)   sk1       (a1: p0)   NONE           NONE
               sk2       (a1: p0)  (r1, p1)   C1
bindx(sk1, add, addr2)   sk1  (a1,a2: p0)     NONE           NONE
               sk2       (a1: p0)  (r1, p1)   C1
bindx(sk1, del, addr1)   sk1  (a2: p0)   NONE           NONE
               sk2       (a1: p0)  (r1, p1)   C1
listen(sk1)         sk1       (a2: p0)   NONE           NONE
               sk2       (a1: p0)  (r1, p1)   C1
sk3 = accept(sk3)   sk1       (a2: p0)   NONE           NONE
               sk2       (a1: p0)  (r1, p1)   C1
               sk3       (a2: p0)  (r2, p2)   C2

The example was created for the TCP-style interfaces, but the equivalent in
the UDP-style can exist with the introduction of the listen() and the
sctp_peeloff().
Now, is there any problem with this? Is this how bindx() is intended to be
used?

     #2. If bindx() is also designed to work on top of the AddIP/DelIP
protocol, which one of the following is the anticipated behaviour for using
bindx(), Table 2 or Table 3?

Table 2
=======

System Call         Socket    Local Endpoint  Remote Endpoint     Assoc
-----------         ------    --------------  --------------- -----
sk1 = socket(STREAM)     sk1  NONE       NONE           NONE
bind(sk1, addr1)    sk1  (a1: p0)  NONE       NONE
listen(sk1)         sk1  (a1: p0)  NONE       NONE
sk2 = accept(sk1)   sk1  (a1: p0)  NONE       NONE
               sk2  (a1: p0)  (r1, p1)   C1
bindx(sk1, add, addr2)   sk1  (a1,a2: p0)     NONE           NONE
               sk2  (a1,[a2]: p0)  (r1, p1)   C1
[C1 negotiates add a2.]
               sk2  (a1,a2: p0)    (r1, p1)   C1
bindx(sk1, del, addr1)   sk1  (a2: p0)   NONE           NONE
               sk2  (a1,a2: p0)    (r1, p1)   C1
[C1 negotiates del a1.]
               sk2  (a2: p0)  (r1, p1)   C1
listen(sk1)         sk1  (a2: p0)  NONE       NONE
               sk2  (a2: p0)  (r1, p1)   C1
sk3 = accept(sk3)   sk1  (a2: p0)  NONE       NONE
               sk2  (a2: p0)  (r1, p1)   C1
               sk3  (a2: p0)  (r2, p2)   C2

The API semantics in Table 2 allow us to add a new interface to a
system by simply manipulating the listening socket (sk1), without the
need to explicitly modify all the open associations (C1, C2).

Table 3  (Table 2 results simulated with Table 1 API semantics)
=======

System Call         Socket    Local Endpoint  Remote Endpoint     Assoc
-----------         ------    --------------  --------------- -----
sk1 = socket(STREAM)     sk1  NONE       NONE           NONE
bind(sk1, addr1)    sk1  (a1: p0)  NONE       NONE
listen(sk1)         sk1  (a1: p0)  NONE       NONE
sk2 = accept(sk1)   sk1  (a1: p0)  NONE       NONE
               sk2  (a1: p0)  (r1, p1)   C1
bindx(sk1, add, addr2)   sk1  (a1,a2: p0)     NONE           NONE
               sk2  (a1: p0)  (r1, p1)   C1
bindx(sk2, add, addr2)   sk1  (a1,a2: p0)     NONE           NONE
               sk2  (a1,[a2]: p0)  (r1, p1)   C1
[C1 negotiates add a2.]
               sk2  (a1,a2: p0)    (r1, p1)   C1
bindx(sk1, del, addr1)   sk1  (a2: p0)   NONE           NONE
               sk2  (a1,a2: p0)    (r1, p1)   C1
bindx(sk2, del, addr1)   sk1  (a2: p0)   NONE           NONE
               sk2  (a1,a2: p0)    (r1, p1)   C1
[C1 negotiates del a1.]
               sk2  (a2: p0)  (r1, p1)   C1
listen(sk1)         sk1  (a2: p0)  NONE       NONE
               sk2  (a2: p0)  (r1, p1)   C1
sk3 = accept(sk3)   sk1  (a2: p0)  NONE       NONE
               sk2  (a2: p0)  (r1, p1)   C1
               sk3  (a2: p0)  (r2, p2)   C2

Thanks,
Daisy

Daisy Chang
IBM Linux Technology Center
[email-masked]
Tel: (512)-838-4194    T/L: 678-4194
FAX: (512)-838-4663

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 6 Sep 2001 19:34:33 -0700 (PDT)

Comments below.

> System Call         Socket    Local Endpoint  Remote Endpoint     Assoc

Yes, I think the above is the intention of the sctp_bindx() call.
I guess we need to have an example to illustrate this in the doc.  
Thanks.

> System Call         Socket    Local Endpoint  Remote Endpoint     Assoc

Table 3 follows the usual socket semantics that the new socket created 
by accept() is independent after it is created.

Note that since the add/del address mechanism is still not finalized, 
the socket draft does not talk about it.  So depending on the authors 
of the add/del address draft, the actual API to initiate add/del address
is not determined at this moment.

							K. Poon.
							[email-masked]

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Fri, 07 Sep 2001 00:28:30 -0500

Daisy was kind enough to move this discussion from the Linux kernel
SCTP list ([email-masked]) over here to TSVWG.

The issue came up because we need to refactor some of our internal
data structures in preparation for implementing the TCP-style API
along side our existing UDP-style API.

Right now I'm leaning toward saying that bindx() on a socket should be
illegal after we have called listen() on it.  Mind you, this is for
the base case where we do not have the addIP option.

I refer to Table 1.  After the first bindx() notice that we have one
socket with the endpoint (a1, a2: p0), and another socket with an
overlapping (but different) endpoint (a1: p0).  RFC2960 explicitly
defines endpoints on a given host as disjoint.  Is it really worth
while violating this requirement?

Kacheong Poon <[email-masked]> writes:
> Comments below.
> [Daisy Chang writes:] 
> > addr1 = (a1, p0)
> > addr2 = (a2, p0)
> > 
> > Table 1
> > =======
> > 
> > System Call         Socket Local ep   Remote ep Asoc
> > -----------         ------ --------   --------- ----
> > sk1 = socket(STREAM)sk1    NONE       NONE      NONE
> > bindx(sk1, add, addr2)sk1  (a1,a2: p0) NONE     NONE
> >                     sk2    (a1: p0)  (r1, p1)   C1
> > bindx(sk1, del, addr1)sk1  (a2: p0)   NONE      NONE
> >                     sk2    (a1: p0)  (r1, p1)   C1

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 07 Sep 2001 14:21:22 +0200

Hi,

see my comments below.

Best regards
Michael

> while violating this requirement?

Assume that you have only cofigured a1 on your host. Furthermore
Assume that you bind a socket to INADDR_ANY and call accept for that socket.
Now there is an incoming request and you establish an assoc having (a1: p0) as
local addresses.
Now you configure a second interface with address a2.
The next incoming connection request will establish an assoc with (a1, a2: p0)
local
addresses. So you have two endpoints which have non disjoint addresses without
uasing the bindx call.

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Fri, 07 Sep 2001 09:03:26 -0500

OK, this is a compelling example, so I think I need to buy into the
semantics in Table 1.

This suggests that the correct semantics for bindx() in the presence
of addIP is Table 3--you need to call bindx() on each socket
separately to add addresses rather than only on the listening socket.

---

From: Jon Grimm <[email-masked]>
Date: Fri, 07 Sep 2001 10:43:02 -0500

In light of the previous discussion, is this the new table (with
addIP/delIP)?

Table 3

			sk2  (a1,[a2]: p0)   NONE       NONE
sk3 = accept(sk1)   	sk1  (a2: p0)  	    NONE        NONE

If so, what state is 'sk1' in after a listen followed by a bindx()?   I
notice that there is a second listen() call in the above table and
wondered if this is required.  

Regards, 
Jon Grimm

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Fri, 07 Sep 2001 12:02:59 -0500

I'm sorry--I should have thought to remove tabs when I built these
tables.  I've removed them here.  There is an error in Table 3,
corrected below.

> bindx(sk2, add, addr2)sk1   (a1,a2: p0)     NONE       NONE

> wondered if this is required.  

If we take sk1 out of the LISTENING state when we call bindx(), then
we could have a short interval where we would have to reject incoming
INIT packets for sk1 arriving to a1.

I think maybe the second listen() should NOT be required.

> > > Now there is an incoming request and you establish an assoc having (a1: p
0) a
> > s
> > > The next incoming connection request will establish an assoc with (a1, a2
: p0
> > )
> > > local
> > > addresses. So you have two endpoints which have non disjoint addresses wi
thou
> > t
> > > > > > The example was created for the TCP-style interfaces, but the equiv
alen
> > t in
> > > > > > Now, is there any problem with this? Is this how bindx() is intende
d to
> >  be
> > > > > > protocol, which one of the following is the anticipated behaviour f
or u
> > sing
> > > > > Table 3 follows the usual socket semantics that the new
> > > > > socket created by accept() is independent after it is created.
> > > > >
> > > > > Note that since the add/del address mechanism is still not
> > > > > finalized, the socket draft does not talk about it.  So
> > > > > depending on the authors of the add/del address draft, the
> > > > > actual API to initiate add/del address is not determined at
> > > > > this moment. 
> > > > >                                                       K. Poon.
> > > > >                                                       [email-masked].
com

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 7 Sep 2001 12:25:27 -0700 (PDT)

> while violating this requirement?

Hmm, I thought we had talked about this before when changing the draft...
If you treat a socket as an endpoint, then according to the RFC 2960, we 
cannot even have accept() (or sctp_peeloff() for UDP socket).  Once a new
socket is created by accept(), it shares exactly the same local address(es)
and port number as the listener socket...  Well, La Monte agreed that we 
could have accept() (-:

The catch is that talking about an endpoint does not really make much 
sense without relating to the association it belongs to.  Once we put 
an endpoint in the context of an association, everything is clear.  The
listener socket does not share the "same address(es)" as the newly created
socket as it is not belonged to any association.  We need to talk about
addresses of both ends of an association and port numbers when talking 
about an endpoint.  So a listener can keep on accepting new associations
as long as the remote address(es) or port numbers of those new requests
are different from the existing ones.  So I don't see why Table 1 posted
by Daisy should be made illegal.  It is the intention of sctp_bindx()
to allow that to happen.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 7 Sep 2001 12:33:19 -0700 (PDT)

> I think maybe the second listen() should NOT be required.

Oops, I missed the second listen() call.  The existing socket semantics
is that once listen() is called, the socket will continue to accept
new associations.  scpt_bindx() does not change this.  A second listen()
call can be used to change the backlog parameter.  It is not needed
to re-activate the listening state of the socket.

							K. Poon.
							[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Mon, 10 Sep 2001 06:44:29 -0500

La Monte/Jon:

Without digging to deeply into the tables (I need to take
the time to do so ...sigh... ) I want to verify that your
comments are not towards someone who as bound IN_ADDRANY...

If I have add-ip in combination with the ANY address binding
then there should NEVER be a need to call sctp_bindx() by
the app. In theory the add-ip/del-ip should just happen when
the interface gets added/deleted...

Only if you do specific binding... i.e.

bind(IP-A);
...
then later do a 

sctp_bindx(IP-NEW-ADDR);

do your semantics below apply... 

Of course this does bring out another issue... how does the app
figure out that a new interface is added... this is why I put
the "read only" routing socket interface in my BSD stuff...
Hmm.. maybe I do need to submit that read-only interface back
to the BSD world... I have avoided it since I thought it was just
a temporary thing.. but maybe not :/

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 10 Sep 2001 13:57:00 +0200

Randy,
just a question for clarification:

If a host has addr-1 and addr-2 and binds a socket to INADDR_ANY.
Now it establishes an assoc. So addr-1, addr-2 will be used in
the INIT or INIT-ACK chunk.

Now addr-3 is configured at the host.

In my understanding the application has to detect this new address
(for example using a routing socket) and calls bindx(addr-3, ADD_ADDR).

Do you mean, this is done automatically by the ADD/DEL supporting 
SCTP implementation?

Best regards
Michael

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Mon, 10 Sep 2001 07:01:51 -0500

> a temporary thing.. but maybe not :/

Shouldn't this be just another notification?  Why isn't 
SCTP_PEER_ADDR_CHANGE with spc_state == ADDRESS_ADDED sufficient?

> > > > Now there is an incoming request and you establish an assoc having (a1:
 p0) a
> > > > The next incoming connection request will establish an assoc with (a1, 
a2: p0
> > > )
> > > > local
> > > > addresses. So you have two endpoints which have non disjoint addresses 
withou
> > > > > > > The example was created for the TCP-style interfaces, but the equ
ivalen
> > > > > > > Now, is there any problem with this? Is this how bindx() is inten
ded to
> > > > > > > protocol, which one of the following is the anticipated behaviour
 for u
> > > > > >                                                       [email-masked]
n.com

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Mon, 10 Sep 2001 06:57:59 -0500

You are correct.  All of these examples are either TCP-style or
UDP-style sctp_peel_off()d sockets with specific interfaces bound.

We haven't mentioned it, but I think the regular UDP-style semantics
are clearer.

 p0) a
a2: p0
withou
ivalen
ded to
 for u
n.com

---

From: Randall Stewart <[email-masked]>
Date: Mon, 10 Sep 2001 08:47:30 -0500

Michael:

If you take a look at the 4.4 ref implementation for FreeBSD
this is precisely what happens. The library is aware of
the routing changes and automatically does add/delete and possibly
set-primary's (when the previous primary is deleted).

This same behavior I will build into our KAME-SCTP implementation
(true kernel land code). Of course we are a ways away from
getting this done but making progress :>

R

---

From: Randall Stewart <[email-masked]>
Date: Mon, 10 Sep 2001 08:51:32 -0500

La Monte:

You I think miss my subtle point here. You get the PEER_ADDR_CHANGE
only after adding the address aka sctp_bindx(new-addr)... so
how do you know the new address exists.

The read-only routing socket I added so that when someone
somewhere does a 

ifconfig new-interface newaddress<cr>

The sctp user library gets a notification so it can do the
add-ip request. In the kernel side I am working on it is
not a problem (well I hope anyway) since I can make a
callback (hopefully :>) to notify the SCTP stack a new
address as come into existance... This works for the bound-all
but what about either library implementations OR a app
that is not monitoring its address list....

R

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 10 Sep 2001 10:58:50 -0700 (PDT)

> the interface gets added/deleted...

Just a clarification...  I remembered we have also talked about 
this when updating the draft.  I think the conclusion is that it 
is implementation dependent.  Some may choose to do the above,
some may not.  That's why we don't specific this in the socket 
API draft about binding to INADDR_ANY.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 10 Sep 2001 11:05:56 -0700 (PDT)

> SCTP_PEER_ADDR_CHANGE with spc_state == ADDRESS_ADDED sufficient?

Another clarification...  The issue we are talking about is the
local address addition/deletion.  The above notification is about
peer...  That notification is sent if the other side decides to
change something.  The additiion/deletion of interfaces in the
local system is system dependent and is outside the scope of the 
API draft, unless of course people really really want to have it
defined here...

							K. Poon.
							[email-masked]

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Tue, 11 Sep 2001 07:20:57 -0500

Kacheong Poon <[email-masked]> writes:
> [La Monte H.P. Yarroll writes:]

Oops.  Of course.

I do not see a really good reason to leave the addition/deletion of
local interfaces implementation dependent.  May I suggest that we add
SCTP_LOCAL_ADDR_CHANGE with a field slc_state that takes the same
values as spc_state?  We will probably need a little more in the
notification structure...

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Tue, 11 Sep 2001 07:48:58 -0500

> how do you know the new address exists.

Ah.  OK, I'm quite happy having user-space implementations find out
about new local interfaces in an application-specific manner.

> that is not monitoring its address list....

Yes, Daisy Chang has already added new interface detection to our
code.  We will use this as a trigger for addIP add/delete requests for
sockets bound to INADDR_ANY.

This still leaves the problem of telling applications which use the
kernel implementations that new local interfaces exist.  Hmm...  Why
would an application need to know this?  As far as SCTP is concerned,
a remote endpoint is equivalent to a path, so knowledge of local
interfaces is not necessary (at the application level).

OK, I think I'll withdraw my earlier proposal for
SCTP_LOCAL_ADDR_CHANGE.

This raises an interesting implementation issue.  If a new local
interface is the preferred route for an existing peer address, should
I re-enter slow start?  I have a new path through the network, so Kiwe
really ought to probe it properly...

> > > > > > > Right now I'm leaning toward saying that bindx() on a socket shou
ld b
> > e
> > > > > > Assume that you bind a socket to INADDR_ANY and call accept for tha
t so
> > cket.
> > > > > > > > > the UDP-style can exist with the introduction of the listen()
 and
> >  the
> > > > > > > > >      #2. If bindx() is also designed to work on top of the Ad
dIP/
> > DelIP
> > > > > > > > > -----------         ------    --------------  ---------------
 ---
> > > > > > > > > system by simply manipulating the listening socket (sk1), wit
hout
> >  the
> > > > > > > > > -----------         ------    --------------  ---------------
 ---
> > > > > > > > Table 3 follows the usual socket semantics that the new socket 
crea
> > ted
> > > > > > > > Note that since the add/del address mechanism is still not fina
lize
> > d,
> > > > > > > > the socket draft does not talk about it.  So depending on the a
utho
> > rs
> > > > > > > > of the add/del address draft, the actual API to initiate add/de
l ad
> > dress
> > > > > > > >                                                       kcpoon@en
g.su

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 11 Sep 2001 18:00:47 -0700 (PDT)

> notification structure...

There are some semantics issues here.  First, when will this notification
be sent to an app?  If an app just bind() to a single address and a new
interface is added, should the app get that notification?  How about
bind(INADDR_ANY) but the app does not want to have the new address
added?

Note that we are talking about addition/deleting of an interface.  Right
now, different implementations (OSes) may have different ways to notify 
apps about this kind of changes.  And this has nothing to do with SCTP.  To
name it SCTP_* is not really appropriate.  Well, unless  of course the
semantics is that those apps bound to INADDR_ANY and the implementation
chooses to implement the  automatic addition/deletion of SCTP local 
addresses will get this notification.  But this seems quite restricted.

So we need to think about several things... 

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Tue, 11 Sep 2001 18:11:26 -0700 (PDT)

> OK, I think I'll withdraw my earlier proposal for

La Monte, you just keeping changing your mind (-:  There may be many
reasons to have this kind of notification for SCTP apps.  One simple 
example can be like this.  A new interface is added and an app finds 
that out.  But it is smart enough to check that the new address is in
the same network as the old address.  And assuming that the app does 
not care about simple interface redunduncy, it does not want to add the
new address as it does not buy any path redunducy.

Note that I assume one semantics of this kind of address change
notification (and its name should not be SCTP_*).  As I mentioned in
a previous mail, there are many issues to be thought about...

> really ought to probe it properly...

??  Isn't it true that whenever SCTP sends to a new address, it has
to slow start?

							K. Poon.
							[email-masked]

---

From: Randall Stewart <[email-masked]>
Date: Wed, 12 Sep 2001 08:48:19 -0500

Comments below :>

> a previous mail, there are many issues to be thought about...

I agree with you Kacheong.. we need to think on this one :>

> ??  Isn't it true that whenever SCTP sends to a new address, it has

Of course it would, the new destination that I add would end up getting
a cwnd=2*MTU and a ssthresh="an arbitrary high value". Thus putting
the destination in slow start (where most of the alternate address
should be camped actually).

R