Subject: [rddp] WG chair on buffer exhaustion

From: [email-masked]
Date: Mon, 21 Jul 2003 15:02:43 -0400

The heated debate on whether lack of availability of a buffer for
an untagged message causes immediate termination of the DDP stream
on which it arrived caused some additional email behind the scenes
in addition to what's been on the list.

With my WG chair hat on, here is what I believe to be the correct
resolution of this issue:

The request to do something other than immediately terminate
a stream when a message shows up that does not have a pre-queued
untagged buffer to receive it does not require that DDP tell
TCP not to ACK the message; an implementation could choose to
do that based on inter-layer implementation optimizations,
but had better do so very rarely.  Alternatively, a DDP
implementation could choose to have some "hidden" resources
in its back pocket that might involve a less efficient receive
path (e.g., receive the errant message to a hidden kernel buffer
and then copy to an application buffer when it becomes available)
- again, this would need to be a rare occurrence for obvious reasons.
This sort of approach is more robust than having TCP not ACK, as
it avoids propagating a host problem into the network and puts the
performance penalty for the implementation problem (should
have had a buffer available, but didn't) squarely on the system
that had the problem.

I would prefer to see the "hidden [kernel] buffer" approach
described as an example rather than the "TCP doesn't ACK" approach.
Among the other problems with "TCP doesn't ACK" is that it creates
a nasty head-of-line blocking problem that affects all traffic
on the connection, not just the untagged traffic that is facing
the resource shortage.

OTOH, if the ULP has set a hard limit on the number of untagged
messages that can come in from a specific stream, and that stream
exceeds the limit, this is evidence that the counterparty isn't
following the rules, and termination of the stream is a reasonable
response.  In the particular scenario that John raises below, the hard
limit would need to be set high enough to accommodate error/special/
emergency conditions, and statistical/shared buffer provisioning would
probably be used to avoid having to provision buffers for simultaneous
occurrence of error/special/emergency conditions on all inbound streams.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> >
> >On Wednesday, July 16, 2003, [email-masked] wrote:
> >
> >>Again, I think that Mallikarjun stated things correctly.  There are
> >>important applications which can only correctly predict the number of 
> >>buffers needed in non error/special/emergency conditions.  When 
> >>error/special/emergency conditions occur, the number and urgency of the 
> >>handling is problematical.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: David Robinson <[email-masked]>
Date: Mon, 21 Jul 2003 14:33:44 -0500

I think that David has made a useful observation here. Having
the protocol specify that the DDP stream be terminated when
an untagged message is presented to DDP when there is a
lack of an available buffer is a reasonable specification.
What it doesn't prohibit is an implementation from
performing tricks to delay delivery which would
result in the stream termination.

Tricks to delay delivery can include having the transport
have a set of hidden buffers, have the transport intentionally
fail to ACK or close its window, or even have the bottom
layer of DDP have hidden buffering that delays delivery.

In a nicely layered approach the semantics are well defined,
if you deliver an untagged message when there are not
buffers available, the DDP stream MUST be terminated. But
in a complex or sophisticated implementation the lower
layers are free to perform whatever tricks it wants to
delay delivery until buffers are available to prevent
termination, but that is a quality of implementation issue.

I would venture that it is not worth a discussion of the
advantages and disadvantages of such trickery as failing
to ACK. Simply state that lower layers MAY delay delivery
of an untagged message to DDP when no buffers are available,
details are left to the implementation.

	-David

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 21 Jul 2003 14:46:13 -0500

> details are left to the implementation.

To the degree that it is an implementation tradeoff, that is true.
But I believe it is relevant to simply state along the lines that
David has suggested that false drops have an impact on the
entire network, and that this solution should be kept to an
absolute minimum.

It is also important that we recognize that the ULP MUST have
some form of flow control, and that *if* that information has
been shared with the DDP Layer via the local interface that
the DDP Layer not only MAY, but MUST terminate the stream
when it receives an untagged segment with an out-of-range
MSN. Pat Thaler's comments last week provide some solid
reasons as to why it would be a bad idea to *mandate* that
this information MUST be shared.

I still think it's worthy of a Must Implement, but there were
valid application models raised where the local ULP
would not want to be bothered with communicating
the limit to the DDP layer.

Caitlin Bestler - [email-masked] - http://asomi.com/

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 29 Jul 2003 12:39:43 -0700

David,

Sorry it took me a while to respond to this discussion.  Also,
I will only have sporadic access to email for the rest of the week.

I guess I generally agree with the direction you're suggesting.
I had earlier suggested certain DDP text changes to address
this issue.  Based on your input, here's one more attempt to 
address the same.  

I hope this is acceptable to everyone since most seem to agree 
that the current DDP text is too restrictive/severe in this area.

Regards.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

The current relevant text is:
section 3.2 (Protocol Overview),
    " Note, however, that
        the ULP needs to address flow control issues because if a DDP
        Message arrives for an Untagged Buffer without an associated
        receive ULP buffer, the DDP Message is dropped, the DDP Stream
        is disabled for reception, and an error is reported to the ULP
        at the Data Sink."

and section 9.1  (Errors detected at the Data Sink).
    "   2.  The QN and MSN have an associated buffer that allows Placement
       of the payload."

My suggestion is to change the text along the following lines:

Section 3.2 to:
    " Note, however, that the ULP needs to address flow control issues.  Any 
      ULP that runs on DDP SHOULD consider the lack of an associated receive 
      buffer to place an inbound Untagged Message of the ULP as an exception."

and section 9.1 to:
    " 2. The QN and MSN have an associated buffer that allows Placement of the 
           payload. In order to gracefully deal with transient buffer exhaustion, some
           DDP implementations MAY choose to consider lack of an associated receive
           buffer to be not an error , such implementations MUST however ensure
           that the affected DDP Segment is eventually recovered in an implementation-specific 
           (as an example, by relying on DDP private buffers) way that guarantees 
           ULP-transparency.  However, the ULP SHOULD NOT rely on the DDP 
           layer gracefully handling this situation on a regular basis."

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 29 Jul 2003 15:51:49 -0500

> " Note, however, that the ULP needs to address flow
> control issues. Any ULP that runs on DDP SHOULD
> consider the lack of an associated receive buffer to
> place an inbound Untagged Message of the ULP as an
> exception."
> 
I would suggest the following section:

Section TBD ULP Flow-control

    The ULP is responsible for providing flow control for
    untagged messages. The Data Source MUST know that it
    is authorized to send an untagged message before sending
    it. The Data Sink MUST know the maximum number of untagged
    messages that it has authorized for reception on a DDP
    Stream or set of streams.
    
    The method by which the Data Source and Data Sink ULP
    reach agreement on this limit is outside the scope of
    this specification. Possible methods include dynamically
    negotiated credits, session negotiated credits and
    fixed limits imposed by the ULP specification.
    
    The Data Sink SHOULD provide sufficient buffers for
    the untagged messages that it has authorized. However,
    it is not required to do so by pre-committing a buffer
    for each valid QN/MSN combination. However the local
    interface MUST allow the Data Sink ULP to pre-specify
    the buffer for each valid QN/MSN combination.
    
    Receipt of an untagged message that was not authorized
    by the Data Sink ULP SHOULD result in termination of the
    DDP Stream. The local interface may allow authorization
    of untagged messages for a specific stream and/or for
    a group of streams.
    
    Receipt of an untagged message that was authorized, but
    for which there is no available buffer, MAY result in
    termination of the DDP Stream. However, implementations
    MAY treat the absence of a buffer for a valid untaged
    message as a local fault. Implementation specific
    procedures to recover from such a fault MAY be used,
    especially if they avoid impacting the remote peer.

I believe this specifically takes care of one iSER concern.
The bulk of the allowed untagged messages have CmdSNs, and
are flow controlled for a *set* of streams that belong to a
single session. It would require that a limit be set for
asynchronous messages, but would allow that to be a ULP
constant rather than a dynamically negotiated value.

It also allows for strict and immediate termination of
untagged messages that are bound to simple Receive Queues
or to the RDMA Read Engine. In the latter case, a "soft
limit" approach would seem to be very inappropriate.

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 30 Jul 2003 12:37:47 -0700

I recall several on this WG have agreed that the
current DDP text that I pointed out is severe.
Let's just fix that text to address the WG concerns.  

You had earlier argued that DDP text is fine the way
it is, now you're proposing a  _lot of_ text that
does not fix the text the WG had a problem with.

No amount of wordsmithing that does not fix the original
text will resolve the issue I had brought up.  Besides, I do 
not agree with all the text you suggest.

Needless to say, I prefer the text I recommended.
--
Mallikarjun

Mallikarjun Chadalapaka

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 30 Jul 2003 15:00:09 -0500

> Needless to say, I prefer the text I recommended.

The text you proposed fails to distinguish between ULP
authorized messages and ULP unauthorized messages.

As has been discussed on the WG, this creates a severe
security vulnerability.

The wording I propose makes it clear that the receiving
side has flexibility in handling a problem caused by
lack of buffering *without* creating a responsibility
to try to recover from unauthorized untagged messages.

Can you be specific as to which portion you disagree
with? And are those disagreements consistent with
keeping iWARP a *reliable* protocol with ULP flow
control?

The bottom line is that when the Data Source sends
an untagged message it is either authorized by the
ULP or it is not.

If it is authorized, the Data Sink is expected to
receive it. Pretending not to receive it, or dropping
the connections are strategies for fault-containment.
That does not mean they are acceptable protocol
behaviors. The wording you propose endorses such
strategies.

If it is not authorized, the Data Sink SHOULD reject
it. It definitely MUST NOT place another session
at risk by accepting an unauthorized message.

The consensus on the WG is *solely* to allow a local
implementation to have some flexibility in how to
recover from fault conditions. Your proposed
wording changes far exceed that.

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 30 Jul 2003 15:46:49 -0500

>       buffer to place an inbound Untagged Message of the ULP as an 

If the local interface has allows the DDP endpoint to know the valid
range of MSNs on a stream (which it will for simple Receive Queues at
the minimum)\ then it definitely SHOULD NOT consider the lack of a
buffer for an *invalid message* to be "an exception".

It is a *violation* of the Upper Layer Protocol, and quite possibly
represents an *attack*. It definitely SHOULD terminate the stream
promptly.

I am willing to accept Pat Thaler's argument that some applications
will find it awkward to share credits with the DDP layer on a per-stream
basis. That is why there is an "If" on the above wording. But the change
you propose implies that the DDP layer SHOULD be attempting to accept
untagged messages by whatever means possible and only terminate the
stream as a last resort.

That would be a mistake, and create a major security vulnerability.

The DDP layer SHOULD reject messages known to be invalid, whether or
not there are buffers available.

I would go further to state that DDP MUST NOT accept invalid untagged
messages if doing so would endanger any other ULP session.