Subject: RE: [rddp] WG chair on buffer exhaustion

From: "Culley, Paul" <[email-masked]>
Date: Tue, 22 Jul 2003 08:29:53 -0500

Do I read into this that you believe that DDP implementations MUST
implement a per stream hard limit on untagged messages ("termination is
a reasonable response")?

If you do, the next question is how fast?  Is it ok to delay the stream
termination under some conditions, until some "extra" untagged messages
have arrived and used buffers?  This might occur if the mechanism for
detecting the hard limit exception depended on counting completions in
software, for example, or if it depended on software to respond to an
exception.

As pointed out earlier in this thread, the mechanism that returns
"credits" for the "hard limit" can become an additional burden for
implementations, depending on how that is done.  A "loose" credit check
and/or update can significantly reduce that burden.  

It is not clear how much guidance the DDP security specs need to provide
on this issue, other than to say something to the effect of: "Here is an
issue, make sure that your implementation, or upper interface and ULP
can deal with it."  Seems like getting too detailed may provide
unnecessary restrictions on implementations.

Paul R. Culley
HP Senior Fellow
281-514-5543

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> Sent: Monday, July 21, 2003 2:03 PM
> To: [email-masked]
> Subject: [rddp] WG chair on buffer exhaustion
> 
> 
> The heated debate on whether lack of availability of a buffer 
> for an untagged message causes immediate termination of the 
> DDP stream on which it arrived caused some additional email 
> behind the scenes in addition to what's been on the list.
> 
> With my WG chair hat on, here is what I believe to be the 
> correct resolution of this issue:
> 
> The request to do something other than immediately terminate
> a stream when a message shows up that does not have a 
> pre-queued untagged buffer to receive it does not require 
> that DDP tell TCP not to ACK the message; an implementation 
> could choose to do that based on inter-layer implementation 
> optimizations, but had better do so very rarely.  
> Alternatively, a DDP implementation could choose to have some 
> "hidden" resources in its back pocket that might involve a 
> less efficient receive path (e.g., receive the errant message 
> to a hidden kernel buffer and then copy to an application 
> buffer when it becomes available)
> - again, this would need to be a rare occurrence for obvious 
> reasons. This sort of approach is more robust than having TCP 
> not ACK, as it avoids propagating a host problem into the 
> network and puts the performance penalty for the 
> implementation problem (should have had a buffer available, 
> but didn't) squarely on the system that had the problem.
> 
> I would prefer to see the "hidden [kernel] buffer" approach 
> described as an example rather than the "TCP doesn't ACK" 
> approach. Among the other problems with "TCP doesn't ACK" is 
> that it creates a nasty head-of-line blocking problem that 
> affects all traffic on the connection, not just the untagged 
> traffic that is facing the resource shortage.
> 
> OTOH, if the ULP has set a hard limit on the number of 
> untagged messages that can come in from a specific stream, 
> and that stream exceeds the limit, this is evidence that the 
> counterparty isn't following the rules, and termination of 
> the stream is a reasonable response.  In the particular 
> scenario that John raises below, the hard limit would need to 
> be set high enough to accommodate error/special/ emergency 
> conditions, and statistical/shared buffer provisioning would 
> probably be used to avoid having to provision buffers for 
> simultaneous occurrence of error/special/emergency conditions 
> on all inbound streams.
> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> 
> > >
> > >On Wednesday, July 16, 2003, [email-masked] wrote:
> > >
> > >>Again, I think that Mallikarjun stated things correctly.  
> There are 
> > >>important applications which can only correctly predict 
> the number 
> > >>of buffers needed in non error/special/emergency 
> conditions.  When 
> > >>error/special/emergency conditions occur, the number and 
> urgency of 
> > >>the handling is problematical.
> 
> _______________________________________________
> rddp mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/rddp
> 

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 22 Jul 2003 08:58:42 -0500

> and/or update can significantly reduce that burden.

As covered earlier on the list, I disagree that this is an unreasonable
burden on the iWARP implementation. The only thing required is a simple
credit count for each DDP stream endpoint

However, Pat Thaler's post did raise legitimate scenarios where 
maintaining
such a count would be an imposition on the local ULP. While I do not 
believe
they represent a typical strategy, they are valid.

So my suggestion is that the drafts move in the direction of being 
permissive
as to the *mechanism* by which the limit is enforced. The security 
draft should
note that use of a "soft" mechanism requires the local ULP to account 
for that
softness and have sufficient slack to absorb the delay. I'll point out 
that
a "soft" credit check combined with sequential ordering can require 
absorbing
a *lot* of slack.

But if a given local interface *can* work, then there is no reason not 
to allow it.
Security drafts should provide guidance on how to evaluate the impact 
of a specific
interface without enumerating them. Focusing on the two questions of 
*when* an invalid
MSN is detected and when a buffer is bound to an MSN provides this type 
of analysis
tool that should be applicable to any local interface.

The goal should remain that accepting an untagged message with an 
invalid MSN should
never put handling of a valid untagged message at risk -- especially 
when the valid
untagged message is part of a different session. (A "session" for these 
purposes is
a set of DDP Streams that serve the exact same set of masters. That is, 
if a fault
on Stream X impairs Stream Y, there is no party that has a complaint 
because at the
application layer Stream X and Stream Y do not represent distinct 
interests.
Generally, a session is a single stream.)

But I believe that goal can be stated without prescribing *specific* 
methods of
achieving it.

Caitlin Bestler - [email-masked] - http://asomi.com/