Subject: [Tsvwg] SCTP Socket API

From: Randall Stewart <[email-masked]>
Date: Thu, 22 Dec 2005 07:28:18 -0500

All:

We have had a LONG thread discussion of the socket API changes
that have been suggested. Here is a summary of the changes
(At least I think) that have been proposed:

A) SOCK_DGRAM as an option for the socket creation besides
    SOCK_STREAM and SOCK_SEQ_PACKET. A SOCK_DGRAM descriptor would
    be one that:
    1) Is fully reliable.. aka its NOT UDP, msg's are delivered
    2) Is atomic, i.e. if I do a read() of 100 bytes and a
       200 byte message is in queue to me, the data would
       be truncated and I would loose the other 100 bytes of
       the message aka no MSG_EOR flag.

B) A new set/get option, SCTP_MAXMSGSIZE. This option
    would set a point in a message size where the stack
    would discard the rest of a fragmented message when
    the size is exceeded... gee I did not say that very
    well ... basically the idea is:

    senderA                                receiverB (
                                           maxmsg set at 4000)

    --------TSN=1 B bit set, 1000 bytes----->
    --------TSN=2,no bits, 1000 bytes------->
    --------TSN=3,no bits, 1000 bytes------->
    --------TSN=4,no bits, 1000 bytes------->
    --------TSN=5,no bits, 1000 bytes------->
    --------TSN=6,E bit set, 1000 bytes----->

    Now when receiverB gets TSN 1, 2 and 3, it
    does its normal thing, but when number 4
    arrives without the E bit set, it delivers
    the 4000 bytes to the upper layer, sets some
    sort of MSG_TRUNC flag, maybe we need a
    SCTP_MSG_TRUN_TOBIG.. or something?? and then
    leaves a dead marker in place so that when 5
    and 6 arrive they are immediately freed and
    discarded as if they had been read.

    For the new SOCK_DGRAM the default would be that
    the SCTP_MAXMSGSIZE is 1/2 the initial
    socket receive window. For SOCK_STREAM and
    SOCK_SEQPACKET the default would be 0, which
    indicates there is NO LIMIT to how big the
    message would be.

C) new socket option SCTP_FRAGMENT_INTERLEAVE. This
    would be at a minimum a gettable option that would
    tell the user if the stack interleaves messages
    from different streams and/or associations. So
    in other words, if a partial delivery is started
    on stream 1, and a message arrives on stream 2 that
    can be delivered. Will the stack send up the message
    on stream 2 while its waiting for more data (a retransmit
    say) on stream 1. The user, if the value returned from
    this is 1, must make sure and read the ancillary data
    on every message... or else it might have an issue
    with mixing up a message from stream one vs stream two :-d
    If the stack allows this option to be set, then it enables/disables
    this behavior.

D) new socket option SCTP_PARTIAL_DELIVERY_POINT. Note that
    this was suggested off-list... This value would be a
    message size that the application would like partial
    delivery to begin at. Normally the stack decides this
    based on resources and other hueristics. If this value is
    set then the application requests that as soon as that
    much information is received (in order) start delivering the message.
    Setting this value to 0, tells t the stack that it should use
    its judgment as to when to start the pd-api.

Now, I think all of the above are reasonable requests... IMO.. and I
don't have a problem with them being added to the socket api.. however
I would like to hear your opinion.. especially from those who
have not joined the discussion and are responsible for the
stacks out there (Michael, Kacheong, Shridhar .. come to
mind and are in my address book.. others as well that
implement SCTP please respond to this :-D)

If I hear no strong objections... I will see if I can
wordsmith up something for the next socket-api iteration ;-D

R
-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Sridhar Samudrala <[email-masked]>
Date: Tue, 27 Dec 2005 15:55:35 -0800

>        the message aka no MSG_EOR flag.
instead, i guess MSG_TRUNC flag will be set.

>     its judgment as to when to start the pd-api.

I think this should be allowed to be set only if SCTP_FRAGMENT_INTERLEAVE 
is supported and enabled.

---

From: Sridhar Samudrala <[email-masked]>
Date: Wed, 28 Dec 2005 14:57:49 -0800

Michael Tuexen wrote:

> Dear all,
>
> see my comments in-line.
>
> Best regards
> Michael
>
> On Dec 28, 2005, at 12:55 AM, Sridhar Samudrala wrote:
>
> I agree with the MSG_TRUNC flag. However, I could imagine that I want to
> use this functionality when I have some message size limits of the upper
> layer protocol and want to simplify my receive code. This would make  
> it simple
> to just ignore packets which are larger than the maximum size.
> But I do not see that this requires a SOCK_DGRAM. I would like to use  
> the
> 1-to-1 style API but still have this feature. So can we make it a  
> socket option
> for SOCK_STREAM and SOCK_SEQ_PACKET?  I do not know if than an  
> additional SOCK_DGRAM
> is needed...

Yes, i would also prefer a socket option instead of a a new SOCK_DGRAM 
socket type.

> This would be a good option providing the above functionality.
> If the user sets this option, he simply tells the kernel that
> he will never want more bytes. If he provides less byte in a
> receive call, the procedure of A) apply. I do not know if it would
> make sense to send (optionally) also an ERROR chunk to let the peer
> know that part of the message was lost. I'm nor sure that
> it is good that the receiving SCTP side discards (part of) a message
> and SACKs it and the peer thinks that SCTP has delivered it to the
> user.
>
>>>
> Why?

May be we don't need this restriction if we only allow single 
outstanding partial delivery
by default and allow multiple partial deliveries on different 
streams/associations when
SCTP_FRAGMENT_INTERLEAVE is enabled.

---

From: Randall Stewart <[email-masked]>
Date: Thu, 29 Dec 2005 11:24:02 -0500

One comment inline below :->

> Yes, i would also prefer a socket option instead of a a new SOCK_DGRAM 

I think I agree with this.. I would rather have a
SCTP_SOCKET_ATOMIC

option of some sort.. instead of a new SOCK_DGRAM type...

That would then allow someone to setup the 1-1 or the 1-M
any way they wanted...

I would still like to hear Kacheong's thoughts as well.. he
may refine my opinion more :-D

I know he is on Holiday.. so we will have to wait
for any rev of the spect to after the first of the
year... that will give time for anyone else as well (who
is on holiday) to respond too ...

R

---

From: Mark Butler <[email-masked]>
Date: Thu, 29 Dec 2005 21:46:10 -0700

Comments below...

Randall Stewart wrote:

>>> layer protocol and want to simplify my receive code. This would 
>>> make  it simple
>>> But I do not see that this requires a SOCK_DGRAM. I would like to 
>>> use  the
>> Yes, i would also prefer a socket option instead of a a new 
>> SOCK_DGRAM socket type.
>
>

I think that is a good thing, provided you are required to set the 
atomicity before connect() / bind() / sendmsg().  Trying to change it 
after data is queued would be problematic.

I think perhaps there should be a combined socket semantics option that 
should take a structure with the read atomicity, the write atomicity, 
the maximum message size, fragment interleave on/off, and partial 
delivery point.

If write atomicity was disabled, you could do multiple writes per 
message and would use MSG_EOR to end each one.  It is a matter of some 
importance that people be able to send messages larger than the send 
buffer.  sendfile() implementations do not terminate records, certainly 
not mid transfer. You have to do the MSG_EOR yourself, and properly so.

Can FreeBSD change atomicity at the socket level, by the way?

>>>>>     SCTP_MSG_TRUN_TOBIG.. or something?? 
>>>>

I think MSG_TRUNC is fine. A message truncated due to exceeding the 
current maximum message size, but not the buffer size is an odd 
condition that requires no different treatment than one exceeding the 
buffer size.  Either way you know you have only part of the message.

>>>>> and then
I do not think basing the default on the socket receive buffer size is a 
good idea, because it is arbitrary on some systems.  Not to mention that 
by tradition setsockopt(SO_RCVBUF) silently truncates whatever you pass 
without an error.  If there needs to be a small default maximum message 
size, I suggest 8K.  If space is not available socket(2) would fail with 
error ENOMEM.

>>> and SACKs it and the peer thinks that SCTP has delivered it to the
Unfortunately, the peer has no such guarantee without further protocol 
support, or ULP confirmation.  Cumulatively acknowledging a message 
doesn't mean the user ever read it from the socket. RFC2960 says that 
implementations should advance the cumulative ack point whenever TSNs 
are received in order.  If you delayed you would seriously increase the 
sending side buffer requirements.  And of course the receiving side 
might crash right after reading the message anyways.

To do this properly, the application or ULP layer needs to be involved, 
so it can confirm that the message was successfully processed, committed 
to disk, resulted in an error or whatever.  So short of a considerable 
protocol extension, it might be best to leave it to the ULP.  RDMA/DDP 
does its own confirmations, as does SMTP, NFS, SMB, ....

I am curious what anyone uses send failures for - the lack of a send 
failure does not indicate the other end received anything, so what do 
people do with them? 

 - Mark

---

From: Randall Stewart <[email-masked]>
Date: Fri, 30 Dec 2005 08:11:41 -0500

> Can FreeBSD change atomicity at the socket level, by the way?

Yes, it can for SCTP at least .. since freebsd lets
one override the so_recv funcition to redefine
your own.. so does NetBSD.. OpenBSD does not.. so
our KAME stack does maintain an older interface that
is still used there....

The atomicity is all a matter of how the so_receive function
operates.. the default one looks at the protocol options
that are setup in the in_proto.c file.. but once you override
the standard so_receive.. you can do whatever you want... and
in order to worry about fragment interleave and all the
other nice things.. one has to do your own so_receive...

> size, I suggest 8K.  If space is not available socket(2) would fail with 

Hmm.. I think 8k is a bit small for a default... gone are the
days of 64Meg of memory even... most machines are coming
standard at 256M and really 1Gig is getting more and more common...

I would think a minimum of 64k.. but really its a Implementation choice
IMO.. having a guideline is ok.. but it really is up to an
  implementation.
> failure does not indicate the other end received anything, so what do 

There are a couple of different types of failures..

a) Ones where the data was sent.. but not ack'ed
b) Ones where they were never sent.

For the class <b> one KNOWS that one must resend this
to an alternative...

For the class <a> one would need to validate via an
application level audit..

Of course there may be other stuff also needing validation.. it
depends...

R

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 28 Dec 2005 09:47:11 +0100

Dear all,

see my comments in-line.

Best regards
Michael

Why?

---

From: Mark Butler <[email-masked]>
Date: Fri, 30 Dec 2005 17:10:34 -0700

> IMO.. having a guideline is ok.. but it really is up to an

The problem is one-to-many sockets with small default shared buffers.  
Whatever the default is, SOCK_DGRAM socket creation should fail if  
messages of the default maximum size cannot be received atomically.  
Otherwise the stack either deadlocks or aborts when a sufficiently large 
message arrives.  That is why the default size should be constrained by 
the default maximum buffer size.

Send failure notifications:

Having SEND_FAILED notifications return the original message is a 
serious constraint on buffer size.  There should be a socket option that 
says how much data from the head of the message is saved for a send 
failure notification.  Zero should mean none.  ENOMEM should be returned 
if the size isn't practical.  

When a send failure occurs, such data should be "moved" from the send 
buffer to the receive buffer to prevent deadlock. IF there is not enough 
room in the receive buffer, the data should be truncated, and both 
MSG_EOR and MSG_TRUNC set on the final read of the notification.

MSG_EOR / MSG_TRUNC policy:

I suggest that whenever a partial delivery is aborted, the next read 
returns zero bytes, with both MSG_EOR and MSG_TRUNC set.  It is a very 
good way to keep readers from running messages together.  Notable case 
is when a PR-SCTP receiver receives a FORWARD TSN that causes it to have 
to abort a partial delivery.  So MSG_TRUNC is a practical necessity even 
on sockets with atomic delivery turned off.

Because of this problem, PR-SCTP capability should have to be enabled 
with an option at the socket level.  By default, messages would still 
expire before transmission started, but not after (as specified by the 
current draft).  I suggest SCTP_PRSCTP_ENABLED for the option name.

 - Mark