Subject: [Tsvwg] Atomic delivery of very large SCTP messages

From: Mark Butler <[email-masked]>
Date: Mon, 05 Dec 2005 11:08:48 -0700

Several have objected to API support for atomic delivery on the grounds 
that it is unnatural to limit the size of SCTP messages, since the wire 
protocol imposes no such limitation.  Although guaranteeing conventional 
reliable atomic delivery of any message smaller than the association 
receive buffer is trivial, there is an alternative that allows 
guaranteed atomic delivery of messages of arbitrary size.   Virtually 
every modern kernel has asynchronous I/O support.  With this support 
comes the ability to copy directly from kernel space into user space.

A socket read operation would be implemented as follows - if there is a 
message in the socket receive queue, copy and return.  Otherwise queue 
the io request. When the first fragment of a new message arrives, bind 
it to an io request, and copy it directly into the user buffer.  Copy 
all in-order chunks from the same message directly into the user buffer 
as well.  If a fragment arrives before its predecessors, temporarily 
hold it.  When the message is complete, signal request completion.

The advantages of implementation support would be:

   1. No need for large buffers in kernel memory
   
   2. Much better throughput
          - no RTT stuttering as the receive window is opened and closed
          - no unnecessary context switches

   3. Much reduced delivery latency
          - no serialization of message deliveries
          - no waiting for the slowest 1:M sender under load
          - no scheduling delays for each partial delivery
          
   4. No indefinite stalling due to peer failure (1:M)

   5. Much reduced 1:M denial of service vulnerability
          - small, shared kernel buffer practical
          - no trivial partial delivery serialization attacks
          - protect all apps, not just ones with limited message sizes

Such support would allow implementations to support atomic message 
delivery semantics for arbitrary message sizes.  All that is needed is a 
standard way to request atomic delivery.

Comments anyone?

 - Mark

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Dec 2005 15:14:21 -0700

Mark,

Mark Butler wrote:                     (Mon, 05 Dec 2005 11:08:48)
> comes the ability to copy directly from kernel space into user space.

Well, no: all read and receive interfaces copy directly from kernel space
into user space; asynchronous I/O just doesn't block the calling process
while it is doing it.

POSIX Asynchronous I/O is dictated by the read() and write() interfaces.
POSIX sockets can support POSIX Asynchronous I/O, however, it cannot
deliver ancillary data so it is rather useless for multiple streams.  If
you want to "improve" on it, you are back with OpenGroup/IEEE.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Mon, 05 Dec 2005 20:45:27 -0700

>into user space; asynchronous I/O just doesn't block the calling process
No in the sense that I mean.  Conventionally, a kernel thread has to 
place the data on the socket receive queue and then wake up a user 
thread to perform the copy (in privileged mode, of course).  Copying 
from the kernel thread eliminates unnecessary context switches and 
unnecessary holding in the socket recieve queue.   O_DIRECT is a similar 
concept for filesystems.

>you want to "improve" on it, you are back with OpenGroup/IEEE.
There is no need for users to use the POSIX AIO or other user level 
async interfaces.  The async support required is transparent to the 
user.  Blocking reads from different threads can proceed in parallel 
with no API support.  For example, the Linux 2.6 series internally 
dispatches all fd read/write operations through asynchronous interfaces 
and then waits for completion if necessary on the return path.

 -Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Dec 2005 21:35:26 -0700

Mark,

Mark Butler wrote:                                                    (Mon, 05 Dec 2005 20:45:27)
> and then waits for completion if necessary on the return path.

With the read() and write() file operations.  QED.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Mark Butler <[email-masked]>
Date: Mon, 05 Dec 2005 23:32:41 -0700

>With the read() and write() file operations.  QED.
Your interpretation is too narrow.  All Linux sendmsg and recvmsg 
operations are dispatched as I described.  See sock_sendmsg and 
sock_recvmsg in net/socket.c.
 
 - Mark

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Dec 2005 23:58:07 -0700

Mark,

This discussion has deteriorated well beyond the scope of the
sockets draft.  Perhaps it would be better to take it over
to sctp-impl list now.  Just drop tsvwg from replies.

--brian

Mark Butler wrote:                            (Mon, 05 Dec 2005 23:32:41)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/