Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 14 Jul 2003 09:58:44 -0700

I agree with Steph that the level of buffer management detail we're
discussing doesn't exactly belong in an architecture doc.

>The C notation is a terse way to express the model.
>Don't read too much into it.

Perhaps this is worth explicitly noting in the draft.

I have issues with two of Caitlin's proposed "rules" wrt
Untagged Messages.

>-- Untagged messages may only be sent as explicitly allowed by
>   the Upper Layer Protocol. The ULP MUST define some mechanism
>   for granting permission to send Untagged messages.

Let me note that "the ULP" in question here could actually be a
protocol on top of what's traditionally called as "the ULP of the
RDMAP".  As an example, the flow control information and buffer
size expectations could be set by iSCSI, even though it may not
be directly running on top of RDMAP/DDP.

>-- Unauthorized untagged messages SHOULD terminate the DDP Stream.

- I saw this stated in an earlier email as well from you - how does the
  DDP layer determine an Untagged Message as authorized/unauthorized?

- Note that this kind of wording again is leaning pretty close to the
  original text in that it essentially makes the same Stream termination
  recommendation in a strong way.  As I stated at the beginning
  of this discussion, I prefer the DDP spec to *not* make any
  recommendations to the RNICs one way or the other.  We should
  only have "SHOULD/SHOULD NOT"s around desirable ULP
  behavior (suggested text below).

- I prefer some wording as David had implied on the original thread -
  "Any ULP that runs on DDP SHOULD consider the lack of a posted
   receive buffer to place an inbound Untagged Message of the ULP
   as an exception.  Further, the ULP SHOULD NOT rely on the DDP
   layer gracefully handling this situation on a regular basis".
---
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
[email-masked]

  a.. To: Stephen Bailey <[email-masked]>, [email-masked]
  b.. Subject: [rddp] Some Minimal end-to-end Buffer Rules Required
  c.. From: Caitlin Bestler <[email-masked]>
  d.. Date: Sat, 12 Jul 2003 12:58:26 -0500
  e.. In-reply-to: <E19bCIv-0000aX-00@ietf-mx>
  f.. List-help: <mailto:[email-masked]?subject=help>
  g.. List-id: IETF Remote Direct Data Placement (rddp) WG <rddp.ietf.org>
  h.. List-post: <mailto:[email-masked]>
  i.. List-subscribe:
<https://www1.ietf.org/mailman/listinfo/rddp>,<mailto:[email-masked]?subject=subscribe>
  j.. List-unsubscribe:
<https://www1.ietf.org/mailman/listinfo/rddp>,<mailto:[email-masked]?subject=unsubscribe>
  k.. Sender: [email-masked]
On Saturday, July 12, 2003, [email-masked] wrote:

>Caitlin,
>
>Sorry I've been slow responding to your previous message on this
>topic.  I'll roll it into this message.
>
>The executive summary is that you should be picking these bones with
>the protocols, not the architecture.  I think the first paragraph of
>the architecture document makes this clear.
>
>> There are two distinct issues when dealing with the Untagged Message
>> buffer model that must be dealt with seperately: flow control and
>> provisioning of buffers.
>
>The architecture IS NOT defining an interface (API, or a looser Verbs
>notion, or whatever else) for RDDP.  It's only modelling what the
>protocols do, in terms of messages and a set of endpoint behavior
>constraints.  The C notation is a terse way to express the model.
>Don't read too much into it.
>
Perhaps it does not belong in the architecture draft, but that's
where it was in the earlier drafts. The first draft required the
local ULP to authorize untagged reception by posting a buffer.
That was reduced to merely requiring the ULP to authorize untagged
reception, but being silent about where buffers came from, and
then to saying nothing about how untagged receptions are regulated.

But the end-to-end protocol needs for certain rules about buffer
usage to be explicitly stated. I suppose we can debate which draft
they belong in precisely, but that does not mean these issues can
be left totally to each implementation. There are certain required
end-to-end behaviors needed to allow ULPs to work.

I don't believe there is any debate on tagged buffers:

-- Tagged messages may only access local memory to the extent that
   is explicitly authorized by the local ULP.
-- The local interface MUST allow the local ULP to enable any
   contiguous subset of its address space through an STag.
-- The local interface MAY allow other combinations of ULP
   accessible memory to be enabled for access through an STag.
   For example, zero-based buffers or maps formed from physical
   buffer lists.
-- The local interface MUST NOT allow the ULP to enable an
   STag with memory access rights that the ULP does not itself
   already have.
-- The remote peer MUST NOT use an advertised buffer in any
   way that is not explicitly authorized by the Upper Layer
   Protocol.
-- The local ULP MUST use the local interface to enable STags
   for all access rights advertised to the remote peer. It MAY
   enable the STags for more rights, but SHOULD NOT do so if
   that places any other process or client at risk.
-- The DDP Layer MUST terminate a DDP Stream if an incoming
   DDP Segment attempts to use an STag in a way that was not
   enabled by the local ULP.

I believe the following rules are required for untagged buffers

-- Untagged messages may only be sent as explicitly allowed by
   the Upper Layer Protocol. The ULP MUST define some mechanism
   for granting permission to send Untagged messages.
-- Authorization for an untagged message MUST specify a maximum
   message size. Receipt of an untagged message that is larger
   MUST result in termination of the DDP Stream.
-- Authorized untagged messages SHOULD be placed into a ULP
   accessible buffer. The local interface MUST allow the ULP
   to supply its own buffer for each authorized untagged message.
   The local interface MAY alternately allow the buffers to be filled
   from a shared resource.
-- Unauthorized untagged messages SHOULD terminate the DDP Stream.
   Failure to terminate an "over credit" DDP Stream MUST NOT imperil
   the availability of resources for any compliant DDP Stream.
-- The receiving side SHOULD have an adequate supply of buffers to
   handle all authorized untagged messages. However, this does not
   require pre-committment of one buffer for each authorized untagged
   message. Statistical sharing of buffer resources MAY be used.
   However the receiving side MUST NOT use this as a form of
   ex post facto flow control.

With these rules, iWARP has extremely powerful and flexible flow
control that is directly responsive to actual ULP needs. It is
superior to the flow control available to non-RDMA IP ULPs.

However, without these rules, an iWARP ULP is both more fragile
and more of a security risk than a conventional non-RDMA ULP.
There are several critical reasons for this:

-- Transport layer buffers no longer pace the stream

    A principal benefit of RDMA protocols is eliminating dependence
    on transport layer buffering. But it is easy to overlook the
    fact that as hated as any bottleneck is, they also have
    side-effects that may have been beneficial. The side effect
    of being forced to pass-through the narrow capacity of
    transport layer buffering was some flow  control.

    With an RDMA protocol there is no predicting where the next
    segment to arrive will be placed. Therefore it would be incorrect
    to hold it off because *one* of the logical destinations may have
    exhausted its capacity.

    The optimal interface between the LLP and DDP allows the LLP
    to pass each validated chunk/segment/whatever to the DDP as
    soon as it is validated and say, "here put this wherever it goes".

    Such an interface, if unchecked, would allow a remote peer to
    rapidly consume available buffers.

-- The LLP cannot detect violations by the remote peer

    All DDP Segments look the same to the LLP.

-- The ULP cannot detect violations by the remote peer

    Virtually all local interfaces that support RDMA protocols
    use a completion queue to deliver completions of untagged
    messages. Frequently these completion queues are:

    *   Opaque. The ULP cannot knows the contents of the queue
        except by dequeuing them serially.
    *   Shared across multiple DDP Streams.

    The net effect is that the ULP cannot know when an untagged
    message was received. By the time the completion is reaped,
    an indeterminant amount of time will have passed. The remote
    peer may have been granted permission during that time.

    Placing responsiblity for catching "cheaters" who sent one
    or two "extra" untagged messages early on the ULP will virtually
    guarantee that cheating will not be caught. If clients are
    competing for resources, for example when they are all using
    a common file server, it would be to the each of the clients
    advantage to cheat. Or at the minimum, because they are competing
    for resources, any client that has an "off by one" error will be
    far more motivated to fix it if they are sending one *too few*
    requests to the server.

    This could easily escalate into an arms race, where everyone
    assumes that the Server is counting on the clients cheating by
    one, and that playing by the rules would place the client at
    a competitive disadvantage versus other clients.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 12:44:58 -0500

>  DDP layer determine an Untagged Message as authorized/unauthorized?
With a simple non-sharing Endpoint it is done by posting receive
buffers. If the ULP has posted 7 receive buffers, it has enabled
a range of 7 MSNs.

If an incoming MSN is outside of that range it is invalid.

This is very close to IB RC, the main difference is that there
is no RDMAP layer message from the Data Sink to the Data Source
that will cause the Data Source to hold the 8th message. iWARP
is relying upon the ULP to enable the MSN *before* it enables
its peer to send that message.

This is a reasonable expectation. The ULP can enable new MSNs
when it restocks the buffer pool or when it sends the reply
message to its peer. No additional interactions are needed,
but in any event, it is a trivial interaction compared to the
other inteeractions already required on a per-transaction basis.

>- Note that this kind of wording again is leaning pretty close
> to the original text in that it essentially makes the same
> Stream termination recommendation in a strong way.  As I
> stated at the beginning of this discussion, I prefer the DDP
> spec to *not* make any recommendations to the RNICs one way or
> the other.  We should only have "SHOULD/SHOULD NOT"s around
> desirable ULP behavior (suggested text below).
>

There is a very big difference.

An Unauthorized message is one that the peer had no business sending.
It indicates a software fault by the peer, or perhaps just in
communication with the peer.

The lack of a a buffer for an authorized message indicates a local
fault in the stocking of the buffer pool. With some of the scenarios
raised, this shortfall could easily be the result of system thrashing
that was not the fault of local ULP. Requiring the ULP to take
responsiblity for the worst case behavior of the OS and other
applications is an unrealistic requirement for a daemon running
in a multiple-application Unix or Windows environment.

>   layer gracefully handling this situation on a regular basis".

Again, if there is no buffer sharing, then the message is simply
unauthorized. The remote peer is misbehaving, and could even be
launching an attack. There is no reason to maintain the stream.

Consider an Accounts Payable example:

An incoming invoice is valid or invalid, no matter how much cash the
company has on hand to pay it. Even if there is cash, Accounts Payable
should not write a check that has not been authorized by some Department
(the ULP in this case).

But let us suppose the invoice is valid, and that the Department/ULP
has authorized it knowing that it was allowed to do so under its budget.
For reasons that are fundamentally identical, most companies do not
segregate funds of a per department basis, but instead works from
a shared pool of funds.

So when a valid invoice arrives, but there is not enough cash to issue
the check, what do you want Accounts Payable to do? The current RDDP
draft would require the equivalent of treating it as an invalid invoice.
That is probably not the response desired. Getting the funds transferred
and *then* issuing the check is. Pretending not to have heard the
DDP Segment is the equivalent of "sitting on" the invoice.

But the same distinction applies: if the message is not authorized then
it should not be placed, even if there is somewhere to place it. That is
the equivalent of allowing Department X to overspend its budget and then
not having enough cash to pay for something Department Y ordered that
was compliant with its budget. (That's the part in the rules I proposed
that says that accepting an unauthorized message MUST NOT adversely 
effect any other process or DDP Stream).

When it is authorized, you SHOULD have somewhere to put it. Accounts
Payable SHOULD have enough cash on hand to pay authorized invoices.
When it doesn't, there has been a fault of some variety. Minimizing
the damage of such a fault should be encouraged, as long as it does
not imply acceptance of the fault-producing behavior.

---

From: Bill Strahm <[email-masked]>
Date: Mon, 14 Jul 2003 13:49:31 -0400 (EDT)

I don't know how you can even SAY MUST drop the connection.

Where a server may be running a hardware RNIC, a client very well may be
running a software implementation over a normal L2 NIC.  In this case, an
Ethernet frame is received, there are no buffers posted to the NIC, so the
frame is dropped on the floor.  The OS may not even be aware that this has
happened.

Now if you want to remove the ability to run various RDMA protocols over
normal L2 interfaces, requiring a Hardware approach, that is fine.  I
haven't seen this requirement in any of the documentation.

Bill

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 13:56:03 -0500

On Monday, July 14, 2003, [email-masked] wrote:

>Where a server may be running a hardware RNIC, a client very
>well may be running a software implementation over a normal L2
>NIC.  In this case, an Ethernet frame is received, there are no
>buffers posted to the NIC, so the frame is dropped on the
>floor.  The OS may not even be aware that this has happened.
>
>Now if you want to remove the ability to run various RDMA
>protocols over normal L2 interfaces, requiring a Hardware
>approach, that is fine.  I haven't seen this requirement in any
>of the documentation.

Obviously a DDP Segment can be dropped anywhere along the way
and the receiving DDP stack will not know it.

So, allowing the DDP receiver to drop a message because of
insufficient buffering is nothing different than any of the
intermediate nodes dropping it.

But the DDP receiver is the first one that

a) Is capable of knowing whether or not the message was
   authorized. The intermediate routers, switches and Ethernet
   NICs simply do not know what a DDP Segment is. They especially
   do not know an MSN is, and they do not know how many receives
   the destination ULP has authorized.
   
b) Is responsible for accepting final delivery. Keep in mind that
   the DDP layer MUST reset the Stream if the LLP *delivers*
   a DDP Segment to it and the payload cannot be placed. Once
   an LLP ack has been generated there is no option for dropping.
   An optimized LLP / DDP interface (which might be real or just
   a fancy euphemism for rototilling) allows the option of detecting
   the lack of buffer during Placement, but that option is *only*
   open during placement.
   
The key here is not the availability of a buffer -- but the validity
of the received message.

Stating that an unauthorized untagged message should result in the
DDP Stream being terminated is ultimately no different than stating
that an invalid STag should result in a terminatin.

In both cases, the peer had no right to send the message. Period.

The issue is how to recognize that having the DDP receiver drop
a valid DDP Segment that it lacks buffers for can be done without
encouraging the ULP to use this capabilities in abusive ways.

But obviously we have to allow it. Getting back to a de facto layered
model, *not* dropping the Ethernet Frame that has a valid but unpaceable
untagged message will ultimately mean dropping some *unknown* Ethernet
Frame or frames. What if the frames you didn't allow the Ethernet NIC
to capture contained Tagged messages that could have been placed?

But we need to distinquish between validation of DDP Segments and buffer
shortages. And within buffer shortages between those that the ULP could
not realistically have prevented, and those caused by the ULP not
fulfilling its responsibilities. We do NOT want to encourage the 
ULP to be dependent on features that are only plausible in thoroughly
rototilled implementations.

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Jul 2003 15:31:57 -0700

>If an incoming MSN is outside of that range it is invalid.

This is implied knowledge outside the scope of the RDDP specifications.

>that will cause the Data Source to hold the 8th message.

IB has a send credit algorithm built into its transport acknowledgement 
wire protocol which enables it to understand whether placement is possible 
in one sense before it gets to the DDP layer.  In addition, due to IB 
requiring strong ordering at the transport layer, i.e. no out-of-order 
placement is allowed, the implementation is simplified in this regard.

>iWARP is relying upon the ULP to enable the MSN *before* it enables its 
>peer to send that message.

As we developed / implemented IB, we learned quite a bit from the usage 
models. Basically, majority of ULP always provide an advertisement method 
and keep track of the number of outstanding operations allowed at a given 
time.  As such, it was possible to eliminate this from the iWARP wire 
protocol as well as enable a well-layered implementation.

>This is a reasonable expectation. The ULP can enable new MSNs when it 
>restocks the buffer pool or when it sends the reply message to its peer. 
>No additional interactions are needed, but in any event, it is a trivial 
>interaction compared to the other inteeractions already required on a 
>per-transaction basis.
>
>
>An Unauthorized message is one that the peer had no business sending.  It 
>indicates a software fault by the peer, or perhaps just in communication 
>with the peer.

It indicates some form of fault / misunderstanding at either end of the 
communication as both sides can be at fault.

>The lack of a a buffer for an authorized message indicates a local fault 
>in the stocking of the buffer pool. With some of the scenarios raised, 
>this shortfall could easily be the result of system thrashing
>responsiblity for the worst case behavior of the OS and other applications 
>is an unrealistic requirement for a daemon running in a 
>multiple-application Unix or Windows environment.

Requiring the ULP to post buffers prior to advertising they can be targeted 
is not unreasonable if the resources are not shared.  If the resources are 
shared, then the operating environment should be reasonably responsive to 
limit the number of shortages without requiring a rework of the ULP or 
requiring a stream to be shutdown.

>launching an attack. There is no reason to maintain the stream.

I prefer to leave this to an implementation to decide.  I can envision 
implementations where the timing may just be off for posting fresh buffers 
and the peers have an implied as opposed to explicit understanding of the 
ULP credits.

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 18:04:01 -0500

On Monday, July 14, 2003, [email-masked] wrote:

>Requiring the ULP to post buffers prior to advertising they can
>be targeted is not unreasonable if the resources are not
>shared.  If the resources are shared, then the operating
>environment should be reasonably responsive to limit the number
>of shortages without requiring a rework of the ULP or requiring
>a stream to be shutdown.

The exact same mechanism can be used when resources are shared.

You post the replacement buffer to the Endpoint/QP. A new MSN
is enabled. If buffers are shared, the buffer is placed in the
shared pool. If buffers are not shared, it is reserved for that
specific MSN (shared buffers are unordered, while stream specific
buffers are ordered).

Doing it this way:

- preserves semantics between shared and unshared. If a message
  is invalid when your peer is fully provisioned, why should
  it become valid because your peer has under-provisioned?
- allows ULPs to manage untagged credits independent of whether
  buffers pooling is in use or not.
- ensure that there is always a simple test on the validity
  of the MSN.
  
The security draft identifies flaws with this sort of shared
resource plan. My objection remains that these are not flaws
of the protocol. Knowing how many messages are enabled at the
receiving QP has clearly been part of the envisioned architecture
all along. The architecture draft went to last call with exactly
that provision until I objected that it was unreasonable to
require that a *buffer* be provided. If you review that discussion,
neither I or anyone else objected to requiring the ULP to let the
local endpoint/QP know how many MSNs it was enabling.

Resource sharing can be achieved without giving up MSN bounds
checking. MSN bounds checking is an important protection against
resources being drained by over-eager clients.

The only thing required is to maintain the "enabled inbound
untagged message" count/credit for each stream -- the same
one that is required for non-sharing endpoints anyway.

A simple interface, more consistent behavior for the ULP designers
to deal with (and someone designing a protocol does not want
to deal with whether a server is using buffer sharing or not),
a simpler implementation, and security holes plugged.

What are you objecting to?

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 14 Jul 2003 18:27:06 -0500

Section 9.1 of the DDP spec, point 5 (page 27):

>   For non-zero length Untagged DDP Segments, the DDP Segment MUST be 
>   validated before Placement by verifying:      
>
... 
>
>   5.  For DDP Messages using Untagged Buffer model, the Message 
>       Sequence Number falls in the range of legal Message Sequence 
>       Numbers, for the queue defined by the QN. The legal range is 
>       defined as being between the MSN value assigned to the first 
>       available buffer for a specific QN and the MSN value assigned to 
>       the last available buffer for a specific QN. 

>From Hemal Shah's presentation at the November 2002 IETF:

>Untagged Buffer data transfer model 
>* Enables data transfer without requiring buffer advertisement 
>* Receiver can queue up a series of ULP buffers to specify the
> order in which the buffers will be consumed (receiver-based
> ULP buffer management) 
>* Each Untagged DDP Message from the Data Source consumes an 
> Untagged Buffer at the Data Sink 
>* Requires associating a receive ULP buffer (based on MSN and
> QN fields) for each DDP Message If an Untagged DDP Message
> arrives without an associated Untagged Buffer, the DDP Message
> is dropped and DDP Stream is terminated 

It strikes me that the concept that the valid range of MSNs
is "unknown" is a revisionist interpretation.

Validation of MSNs was part of the protocol before resource
pooling was proposed, and resource pooling can be achieved
without elimination of MSN validation.

Eliminating MSN validation makes implementation independent
end-to-end ULP flow control more difficult to enforce. It
makes implementations more complex (because the range of
valid MSNs is no longer determined *before* a packet arrives),
and it creates security concerns.

The only change that was proposed was eliminating the need to
specify a specific buffer when enabling an untagged receive.
I maintain that this is the *only* change required.