Subject: Re: [Tsvwg] Atomic delivery of very large SCTP messages

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 05 Dec 2005 23:57:19 +0000

> -----Original Message-----
> From: Brian F. G. Bidulock [mailto:[email-masked]]
> Sent: Monday, December 5, 2005 10:14 PM
> To: 'Mark Butler'
> Cc: [email-masked]
> Subject: Re: [Tsvwg] Atomic delivery of very large SCTP messages
> 
> Mark,
> 
> Mark Butler wrote:                     (Mon, 05 Dec 2005 11:08:48)
> > Several have objected to API support for atomic delivery on the grounds 
> > that it is unnatural to limit the size of SCTP messages, since the wire 
> > protocol imposes no such limitation.  Although guaranteeing conventional 
> > reliable atomic delivery of any message smaller than the association 
> > receive buffer is trivial, there is an alternative that allows 
> > guaranteed atomic delivery of messages of arbitrary size.   Virtually 
> > every modern kernel has asynchronous I/O support.  With this support 
> > comes the ability to copy directly from kernel space into user space.
> 
> Well, no: all read and receive interfaces copy directly from kernel space
> into user space; asynchronous I/O just doesn't block the calling process
> while it is doing it.
> 
> POSIX Asynchronous I/O is dictated by the read() and write() interfaces.
> POSIX sockets can support POSIX Asynchronous I/O, however, it cannot
> deliver ancillary data so it is rather useless for multiple streams.  If
> you want to "improve" on it, you are back with OpenGroup/IEEE.
> 

And I believe the OpenGroup has already decided that putty around
the corners of the sockets API won't solve the problem, a fundamental
shift to a QP/CQ style interface is needed. See the "Extend Sockets API"
from the ICSC (under the OpenGroup).

A QP/CQ style interface naturally delivers messages atomically to
the consumer in the consumer's buffer, and allows the consumer
to control the maximum amount of resources tied up in partial
messages.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Mark Butler <[email-masked]>
Date: Mon, 05 Dec 2005 23:16:56 -0700

Caitlin Bestler wrote:

>> <>
As I imagine you are aware,  Linux has AIO / CQ syscalls already, so an 
ES-API interface could be implemented as a simple user level wrapper 
around existing kernel interfaces.  The internal protocol driver 
interfaces are more than adequate for ES-API support.  All that is 
needed is to upgrade the protocol drivers to handle async operations 
without silently degrading into synchronous mode.  Such an upgrade 
change would benefit users of existing interfaces in the way I have 
outlined.

However, atomic message delivery will not happen unless a protocol 
driver supports it, which means that ES-API with SCTP sockets will be of 
limited value until stacks are upgraded to support atomic deliveries, as 
specified by the ES-API.  Async pipelined socket I/O is practically 
impossible unless you can predict which inbound messages will arrive in 
which posted buffers.  Atomic delivery makes this trivial on reliable 
one-to-one connections.

I am curious why ES-API does not provide for read requests bound to a 
specific stream.  Practically every modern I/O technology provides for 
multiple independent stream pairs - SCTP, Infiniband and RDMA come to 
mind.  Without a way to read from a specific stream, multiple streams 
per association are good for little but packet loss mitigation.  One 
alternative is setting up a bunch of parallel associations, which is 
what SCTP and Infiniband, etc. were theoretically designed to avoid.  I 
guess the simple solution would be a way to peel off stream specific 
sockets, so you can efficiently have one socket per QP.

One problem I have with ES-API is that it is unnecessarily narrow -  
adding several new abstractions that can only be used with sockets.  If 
a new completion mechanism is going to be added, it should work across 
all file descriptors, not just sockets.  File operations could really 
use the improvement as well - signal based completion in a multithreaded 
environment is a bit of a disaster, to put it mildly.  Memory 
registration is something that has broad utility as well.  (And 
requiring a call to exs_init is ugly).  

I don't suppose there are plans for IPPROTO_RDMA sockets are there? with 
one socket per QP?

 - Mark

---

From: Jeremy Harris <[email-masked]>
Date: Thu, 08 Dec 2005 14:11:18 +0000

Mark Butler wrote:
> environment is a bit of a disaster, to put it mildly.

We were constrained by our terms of reference, to only produce an
interface applying to sockets.  This did not stop many, perhaps most,
of the committee agreeing with your point of view and the interface
was designed with the extension to nonsocket use in mind.

- Jeremy