Subject: RE: [rddp] Shared Receive Queues (Re: I-D ACTION:draft-ietf-rddp- arch-02)

From: [email-masked]
Date: Mon, 30 Jun 2003 19:11:22 -0400

> Comparing this draft to the prior, the main change I can see 
> is the removal of the "ddp_post_recv" operation.
> 
> This is a problem.
> 
> First there is a procedural issue. The only issue raised as a 
> result of the last call and post receive was mine. 
> Specifically, I objected to the requirement that a *buffer* 
> be provided with the ddp_post_recv operation, because an 
> implementation might choose to allow the buffers to be taken 
> from a buffer pool.

That's enough to hold both drafts (problem statement and architecture)
for a revision of the architecture draft after Vienna - Caitlin
is correct that last call comments generally need to be resolved
to the satisfaction of the person raising the comment.  The
reason for holding both drafts is that they need to go to the
IESG and IETF Last Call as a pair.

The issue raised is valid and needs to be discussed - it will be
on the agenda for Vienna, but the final resolution will be on this
list.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]] 
> Sent: Monday, June 30, 2003 6:45 PM
> To: [email-masked]
> Subject: [rddp] Shared Receive Queues (Re: I-D 
> ACTION:draft-ietf-rddp-arch-02)
> 
> 
> Comparing this draft to the prior, the main change I can see 
> is the removal of the "ddp_post_recv" operation.
> 
> This is a problem.
> 
> First there is a procedural issue. The only issue raised as a 
> result of the last call and post receive was mine. 
> Specifically, I objected to the requirement that a *buffer* 
> be provided with the ddp_post_recv operation, because an 
> implementation might choose to allow the buffers to be taken 
> from a buffer pool.
> 
> The actual change is far greater than that however. It 
> eliminates the requirement that reception of each untagged 
> message be pre-enabled by the ULP.
> 
> In my opinion this is a grave mistake. It creates robustness 
> and security issues.
> 
> With this change we are seeing a presumption of a solution 
> being written into the draft, after last call, when that 
> proposal has not been made on the reflector.
> 
> 
> Secondly, there is the substantive issue. There is a problem 
> that has been identified, but not properly discussed within 
> the WG. It needs to be discussed, and a concensus reached on 
> the best solution.
> 
> The label some will give to this is "Shared Receive Queues", 
> but that is a label of *one* solution. I also believe it is 
> the wrong solution.
> 
> The problem would be better described as "Over Dedication of 
> Server Buffers".
> 
> Basically, with the presumed local interface described in the 
> first two drafts of the architecture, a server that supports 
> a large number of concurrent clients would have to dedicate 
> for too much memory to receive untagged requests from clients.
> 
> Basically, if each client is allowed to have 2 messages in 
> flight, and there are 100,000 clients, then the server would 
> be required to pre-post 200,000 buffers.
> 
> However, it is quite possible that given the amount of time 
> it takes to process one received request, and the maximum 
> number of requests that can arrive through the server's 
> physical links, that there is never a need for more than 
> 50,000 buffers.
> 
> I believe that there can be full agreement that this is a 
> legitimate problem, and that neither the architecture or wire 
> protocol should preclude a server from offering a local 
> interface with a solution.
> 
> However, I believe this can be solved without eliminating the 
> rule that the Data Sink ULP must pre-authorize each untagged 
> message. Further, eliminating that rule has an adverse affect 
> on the robustness and security of RDMA protocols.
> 
> The shared receive queue solution is described in 
> draft-hilland. It basically allows an endpoint to pull 
> receive buffers from a shared receive queue rather than from 
> its own receive queue. Replacement buffers are pre-posted to 
> the receive queue, rather than to the specific receive queue.
> 
> What I propose instead is allowing a receive queue to be fed 
> from a shared buffer pool. The Data Sink would still 
> pre-authorize each receive for each endpoint. So there would 
> be a trackable credit count for each endpoint/QP. If the 
> endpoint/QP has a shared buffer pool, any buffer supplied 
> would be placed in the buffer pull. Alternately the ULP could 
> pre-authorize a receive without supplying a buffer (thereby 
> under-provisioning the pool), or post a buffer directly to 
> the shared buffer pool (to reduce the under-provisioning).
> 
> Both the Shared Receive Queue and Shared Buffer Pool allow a 
> local interface to optionally enable a server to 
> under-provision the receive buffers supporting a large number 
> of concurrent clients.
> 
> Both methods accept the risk that a DDP Stream will have to 
> be torn down should the Server ULP do a poor job of making 
> its statistical predictions. However this is considered an 
> acceptable risk for server applications to make. Local 
> Interface features to support under-provisioning would never 
> be mandatory-to-implement, and would never be configured as a 
> default option that could be selected by a naive application 
> developer.
> 
> The difference between the two methods is in how the system 
> will respond when one client exceeds it allotted credits. 
> This difference has both robustness and security issues. 
> 
> RDMA-enabled applications are dependent on ULP pacing for 
> flow control. Transport layer buffering is no longer 
> effective. RDMA-servers typically pace their clients by 
> controlling the maximum number of in-flight untagged 
> messages. DAFS negotiates a limit per session. NFS allows for 
> it to be dynamically adjusted.
> 
> In either case, a typical processor loop is as follows:
> 
>     post_initial_receive_buffers()
>     while (get_untagged_message_completion())
>         process_received_request()
>         post_replacement_receive_buffer() -- variable under NFS
>         post_untagged_message_response()
>         
> There are some important characteristics of this loop. It is 
> self-adaptive, the longer it takes to process a request the 
> slower the new message is posted.
> 
> It also works with either Shared Receive Queues or Shared 
> Buffer Pools. The only difference is in where receive_buffers 
> are posted.
> 
> The difference comes when the remote client is over-eager and 
> sends more than it is allowed to, but not enough more that it 
> would drain the shared resources totally.
> 
> With Shared Receive Queues
> 
>     The DDP Stream is NOT terminated.
>     
>     If the client exceeds a configured limit for the stream, a notice
>     will be given to the ULP. But the DDP Stream will NOT be impaired
>     in any way (i.e., the client can exceed the limit even more until
>     the ULP *responds* to the limit).
>     
>     Even if detected, the limit is configured, i.e. static. It does
>     not automatically adjust while a request is being processed.
>     An over-eager client that had an "off by one" error in its favor
>     could rely upon not being caught.
>     
>     If the ULP response is sufficiently slow, a different DDP Stream
>     will fail to obtain a buffer, resulting in termination of that
>     DDP Stream. It is one thing for the Server to take this risk based
>     upon its prediction of statistical buffer usage. If limits are
>     actually hit, the server is probably already in overload state
>     anyway and not providing very good service as it is. This is
>     a different vunerability to a form of DoS attack. This is
>     listed in the security draft, but it is incorrectly identified
>     as being a trait of any RDMA server that shares a completion queue
>     across multiple clients. In fact the vunerability exists only when
>     Shared Receive Queues are used.
>     
> With a Shared Buffer Pool
> 
>     The DDP Stream is terminated as soon as it exceeds its allotment,
>     whether or not there are buffers available.
>     
> 
> Enforcement of ULP flow control is vital to robust and secure 
> RDMA applications. Further, it is not something that the ULP 
> can naturally do for itself. Typically completed receives are 
> queued in a way that is not visible to the ULP. by the time 
> the ULP reaps these completions there is no way to know what 
> the "high water mark" was. Further complicating the local 
> interface to allow for alarms just complicates the local 
> interface, and still leaves a vunerability because illegal 
> receives will continue until the ULP responds. A DoS defense 
> that is dependent upon the ULP reacting to an asynchronous 
> notification has some serious limitations.
> 
> I am not suggesting that Shared Buffer Pools be adopted as an 
> official IETF blessed interface. Rather, I am suggesting that 
> Shared Receive Queues NOT be recognized.
> 
> Shared Buffer Pools have no implications on the definition of 
> the ULP. Shared Receive Queues change the semantics on when a 
> DDP Stream will be terminated, and hence undermine the 
> enforcement of ULP flow control. The original architecture is 
> preserved, with the exception of allowing receives to be 
> enabled without providing a buffer. But buffers are a 
> local issue, they should not have end-to-end implications. 
> Authorization of receive operations is visible to the peer. I 
> am particularly concerned that the failure of some 
> implementations to enforce this rule will in fact create a 
> "rule" where clients *expect* to be able to "cheat" a bit, 
> and will consider those servers that enforce credit limits 
> properly to be defective.
> 
> 
>     
> 
> 
> 
> 
> 
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp