Subject: [rddp] I-D ACTION:draft-ietf-rddp-arch-02.txt

From: [email-masked]
Date: Mon, 30 Jun 2003 07:53:46 -0400

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Remote Direct Data Placement Working Group of the IETF.

	Title		: The Architecture of Direct Data Placement (DDP)And 
                          Remote Direct Memory Access (RDMA)On Internet 
                          Protocols
	Author(s)	: S. Bailey, T. Talpey
	Filename	: draft-ietf-rddp-arch-02.txt
	Pages		: 17
	Date		: 2003-6-27
	
This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports.  This architecture does
not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding
the protocols.  DDP allows the efficient placement of data into
buffers designated by Upper Layer Protocols (e.g. RDMA).  RDMA
provides the semantics to enable Remote Direct Memory Access
between peers in a way consistent with application requirements.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-rddp-arch-02.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-rddp-arch-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-rddp-arch-02.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 30 Jun 2003 17:44:48 -0500

Comparing this draft to the prior, the main change I can see is the
removal of the "ddp_post_recv" operation.

This is a problem.

First there is a procedural issue. The only issue raised as a result
of the last call and post receive was mine. Specifically, I objected
to the requirement that a *buffer* be provided with the ddp_post_recv
operation, because an implementation might choose to allow the buffers
to be taken from a buffer pool.

The actual change is far greater than that however. It eliminates the
requirement that reception of each untagged message be pre-enabled by
the ULP.

In my opinion this is a grave mistake. It creates robustness and
security issues.

With this change we are seeing a presumption of a solution being
written into the draft, after last call, when that proposal has
not been made on the reflector.

Secondly, there is the substantive issue. There is a problem that
has been identified, but not properly discussed within the WG.
It needs to be discussed, and a concensus reached on the best
solution.

The label some will give to this is "Shared Receive Queues",
but that is a label of *one* solution. I also believe it is
the wrong solution.

The problem would be better described as "Over Dedication of
Server Buffers".

Basically, with the presumed local interface described in the
first two drafts of the architecture, a server that supports
a large number of concurrent clients would have to dedicate
for too much memory to receive untagged requests from clients.

Basically, if each client is allowed to have 2 messages in flight,
and there are 100,000 clients, then the server would be required
to pre-post 200,000 buffers.

However, it is quite possible that given the amount of time it takes
to process one received request, and the maximum number of requests
that can arrive through the server's physical links, that there is
never a need for more than 50,000 buffers.

I believe that there can be full agreement that this is a legitimate
problem, and that neither the architecture or wire protocol should
preclude a server from offering a local interface with a solution.

However, I believe this can be solved without eliminating the rule
that the Data Sink ULP must pre-authorize each untagged message.
Further, eliminating that rule has an adverse affect on the robustness
and security of RDMA protocols.

The shared receive queue solution is described in draft-hilland.
It basically allows an endpoint to pull receive buffers from a
shared receive queue rather than from its own receive queue.
Replacement buffers are pre-posted to the receive queue, rather
than to the specific receive queue.

What I propose instead is allowing a receive queue to be fed from
a shared buffer pool. The Data Sink would still pre-authorize each
receive for each endpoint. So there would be a trackable credit
count for each endpoint/QP. If the endpoint/QP has a shared buffer pool,
any buffer supplied would be placed in the buffer pull. Alternately
the ULP could pre-authorize a receive without supplying a buffer
(thereby under-provisioning the pool), or post a buffer directly
to the shared buffer pool (to reduce the under-provisioning).

Both the Shared Receive Queue and Shared Buffer Pool allow a local
interface to optionally enable a server to under-provision the receive
buffers supporting a large number of concurrent clients.

Both methods accept the risk that a DDP Stream will have to be torn
down should the Server ULP do a poor job of making its statistical
predictions. However this is considered an acceptable risk for server
applications to make. Local Interface features to support
under-provisioning would never be mandatory-to-implement, and would
never be configured as a default option that could be selected by
a naive application developer.

The difference between the two methods is in how the system will respond
when one client exceeds it allotted credits. This difference has both
robustness and security issues. 

RDMA-enabled applications are dependent on ULP pacing for flow control.
Transport layer buffering is no longer effective. RDMA-servers typically
pace their clients by controlling the maximum number of in-flight
untagged messages. DAFS negotiates a limit per session. NFS allows
for it to be dynamically adjusted.

In either case, a typical processor loop is as follows:

    post_initial_receive_buffers()
    while (get_untagged_message_completion())
        process_received_request()
        post_replacement_receive_buffer() -- variable under NFS
        post_untagged_message_response()
        
There are some important characteristics of this loop. It is
self-adaptive, the longer it takes to process a request the
slower the new message is posted.

It also works with either Shared Receive Queues or Shared Buffer Pools.
The only difference is in where receive_buffers are posted.

The difference comes when the remote client is over-eager and sends
more than it is allowed to, but not enough more that it would drain
the shared resources totally.

With Shared Receive Queues

    The DDP Stream is NOT terminated.
    
    If the client exceeds a configured limit for the stream, a notice
    will be given to the ULP. But the DDP Stream will NOT be impaired
    in any way (i.e., the client can exceed the limit even more until
    the ULP *responds* to the limit).
    
    Even if detected, the limit is configured, i.e. static. It does
    not automatically adjust while a request is being processed.
    An over-eager client that had an "off by one" error in its favor
    could rely upon not being caught.
    
    If the ULP response is sufficiently slow, a different DDP Stream
    will fail to obtain a buffer, resulting in termination of that
    DDP Stream. It is one thing for the Server to take this risk based
    upon its prediction of statistical buffer usage. If limits are
    actually hit, the server is probably already in overload state
    anyway and not providing very good service as it is. This is
    a different vunerability to a form of DoS attack. This is
    listed in the security draft, but it is incorrectly identified
    as being a trait of any RDMA server that shares a completion queue
    across multiple clients. In fact the vunerability exists only when
    Shared Receive Queues are used.
    
With a Shared Buffer Pool

    The DDP Stream is terminated as soon as it exceeds its allotment,
    whether or not there are buffers available.

Enforcement of ULP flow control is vital to robust and secure RDMA
applications. Further, it is not something that the ULP can naturally
do for itself. Typically completed receives are queued in a way that
is not visible to the ULP. by the time the ULP reaps these completions
there is no way to know what the "high water mark" was. Further
complicating the local interface to allow for alarms just complicates
the local interface, and still leaves a vunerability because
illegal receives will continue until the ULP responds. A DoS defense
that is dependent upon the ULP reacting to an asynchronous notification
has some serious limitations.

I am not suggesting that Shared Buffer Pools be adopted as an official
IETF blessed interface. Rather, I am suggesting that Shared Receive
Queues NOT be recognized.

Shared Buffer Pools have no implications on the definition of the ULP.
Shared Receive Queues change the semantics on when a DDP Stream will
be terminated, and hence undermine the enforcement of ULP flow control.
The original architecture is preserved, with the exception of allowing
receives to be enabled without providing a buffer. But buffers are a 
local issue, they should not have end-to-end implications. Authorization
of receive operations is visible to the peer. I am particularly
concerned that the failure of some implementations to enforce this
rule will in fact create a "rule" where clients *expect* to be able
to "cheat" a bit, and will consider those servers that enforce credit
limits properly to be defective.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp