Subject: [rddp] Review comment on DDP untagged buffer model

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 10 Jul 2003 19:45:19 -0700

I  have a comment on the DDP untagged buffer model
of the current DDP spec.

The DDP spec implies that the DDP Stream is to be
dropped by implementations when there is no receive
buffer to place an inbound untagged message.  This
is mandating a specific type of implementation, and IMHO,
is needless.  The DDP spec should instead allow for
implementation flexibility in this area.  Those RNICs
that can gracefully deal with lack of receive buffers
via LLP means (such as TCP retransmit) should be
allowed as well.  I can agree with the authors' original intent
that ULP-level flow control and/or adequate buffer provisioning
must be done to avoid needless retransmits, but mandating
a connection drop on the first instance of hitting a (most
likely a very transient) no-buffer window is severe.

I believe the DDP spec can allow this RNIC implementation
flexibility with minimal text changes.  My review uncovered
only two areas of text that mandated the connection drop
requirement.

section 3.2 (Protocol Overview),
    " Note, however, that
        the ULP needs to address flow control issues because if a DDP
        Message arrives for an Untagged Buffer without an associated
        receive ULP buffer, the DDP Message is dropped, the DDP Stream
        is disabled for reception, and an error is reported to the ULP
        at the Data Sink."

and section 9.1  (Errors detected at the Data Sink).
    "   2.  The QN and MSN have an associated buffer that allows Placement
       of the payload."

My suggestion is to change the text along the following lines:
Section 3.2 to:
    " Note, however, that the ULP needs to address flow control issues."
and section 9.1 to:
    " 2. The QN and MSN have an associated buffer that allows Placement of the payload. DDP implementations
MAY optionally consider lack of an associated buffer to be not an error, but such implementations MUST ensure
that the DDP Segment is eventually recovered using LLP means in a ULP-transparent way."
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
[email-masked]

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 10 Jul 2003 22:27:03 -0500

There are two distinct issues when dealing with the Untagged Message
buffer model that must be dealt with seperately: flow control and
provisioning of buffers.

Requiring the Data Sink to *always* prepost a buffer to enable each
reception of each untagged message is clearly wasteful. I posted an
objection to the last call on the architecture document on this point
earlier. Requiring pre-committed buffers for all eligible untagged
messages simply takes too much memory. A server could easily have
50,000 clients and allow any one of them to have 6 messages in flight.

With the simple "one buffer - one authorized MSN" rule you would have
to commit 300,000 buffers. But statistics and/or the aggregate bandwdith
of your connection to those 50,000 clients might mean that only 50,000
buffers were needed.

So the protocol needs to allow buffers to be under-provisioned, that is
taken from a common pool, should the local interface choose to offer
such a capability. It certainly would not be a mandatory feature, it
really only applies when you have a large number of connections.

But there is a second aspect - flow control.

Each untagged message needs to be authorized by the Data Sink
through some mechanism. Typically this is based upon a session
established number of "send credits". Each untagged message
from the Data Source consumes a credit. A reverse untagged
message typically restores it (although other ULP mechanisms
could be imagined, the NFS drafts allow the credits to be
adjusted by an amount other than one in the reply, for exmaple).

The question is, who enforces this limit.

The traditional approach is to require the Data Sink to pre-enable
each receive by posting a buffer. I believe that should be relaxed
only to the extent of eliminating the need for a buffer to be supplied.
The Data Sink ULP would still be required to authorize each reception,
but it MAY ask that the buffer come from some specified pool or even
a system pool (at the discretion of the local interface as to whether
such a resource is available, and if available whether this user
can use it).

Requiring the ULP to authorize each untagged message in advance
has several advantages:

-- it allows the receiving endpoint to bound the range of valid
   MSNs simply.  This range is a very critical resource, the EP
   must track the receive status of every message within the
   eligible window. If an errant peer jumps ahead 100 MSNs, then
   the EP has to track the completion status of all 100 intermediate
   messages.
-- If a buffer pool is in fact use, it prevents a single peer from
   draining that pool.
   
Keep in mind that traditional buffer based flow control does not
apply to RDMA streams. This is dealt with in the applicability
draft in greater detail.  But effectively, ULP pacing is more
critical for RDMA applications than for non-RDMA applications
that have the "benefit" of being paced by traditional transport
layer bottlenecks.

Allowing statistical pooling of buffers is undoubtedly of benefit
to many implementations -- and the protocol should clearly allow
this. But I disagree that it is a "benefit" to be forgiving to
remote peers that are sending more untagged messages than they
are allowed. It is actually a performance and security concern.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 11 Jul 2003 11:53:45 -0500

On Thursday, July 10, 2003, [email-masked] wrote:

> " 2. The QN and MSN have an associated buffer that allows
> Placement of the payload. DDP implementations MAY optionally
> consider lack of an associated buffer to be not an error, but
> such implementations MUST ensure that the DDP Segment is
> eventually recovered using LLP means in a ULP-transparent way."

As I indicated previously, this is a tricky area. It is very
analogous to the problem of the MPA layer detecting a CRC-32c
error on material that had already passed the TCP checksum.
It is very easy to say, "declare it a bad packet", but that
will only work if the retry is initiated at the reverse end
above the TCP layer. If your peer's MPA hands TCP segments
to a TCP layer cleanly, you will simply get the same faulty
TCP segment repeated.

The two issues that require care here are:

a) The QN/MSN must still be authorized by the ULP. Allowing
unauthorized indefinite reception presents major security
concerns. Mechanisms that require ULP reaction to being
"overlimit" are inherently suspect as methods of preventing
a Denial Of Service attack. Forcing frequent trips to the ULP
may be a sufficient victory for the attack to succeed.

b) While a "conditional placement" interface from the LLP
is feasible, and even natural coming from MPA/TCP, it is
not something that any ULP should ever rely upon. It is
unnatural for the LLP behavior of the receiver, even a
rototilled one, to be dependent on higher layers.

I won't make that an absolute. I've done network protocol
stacks that included the feature that if the processing
of a received Ethernet Frame caused an exception the
*only* result would be to log the error. Except for the
log, the system would simply pretend the packet had never
arrived. It was a natural and cheap recovery mechanism
that was far preferable to rebooting the machine.

I'm not sure what sort of wording, but that is the type
of "conditional placement" interface that I think would
be acceptable. It is one that *expects* to place the data,
or expects to declare the peer to be in violation. "Try
again later" should only be reserved as a method of recovering
from a system fault. It should never be part of normal
operations.

When that packet is dropped, the peer LLP is going to assume
that it was due to congestion. This will have a major impact.

Indeed, when the LLP is SCTP it would be highly incorrect to
allow *one* DDP Stream that had under-provisioned its buffers
to adversely affect the entire associations congestion window.
If that stream really abused the feature, the peer could flip
the entire association to an alternate path.

An additional factor on this not being something that the
ULP should be relying upon is that "conditional placement"
is *not* a natural LLP interface for an SCTP stack. Because
the SCTP adaptation uses unordered SCTP delivery, the SCTP
layer can simply push received Data Chunks to the DDP layer.
If the pushed chunk cannot be placed, the DDP layer simply
resets the DDP Stream. It has *no* effect on the SCTP layer.
Further, this could be cleanly layered over any SCTP stack
that had an optimized interface for unordered deliveries.

It make senese for the spec to allow the receiving stack some
flexibility in handling a *fault* situation. Not having a buffer
that should have been present for an authorized untagged message
can indeed qualify as a fault. Particularly if the reason it was
not available was due to congestion on the system bus or other
unusual delays that prevented the ULP from restocking a buffer pool.

In response to this type of fault, a false drop is better than a
disconnect. But we need to be very clear that the ULP is
responsible for properly pacing itself. The Data Sink is
expected to be ready to handle all untagged messages that it has
authorized reception of. Dropping packets must never be a method
of retroactive flow control (oops, I'm busier than I thought I
would be). Dropped packets at the transport layer are supposed
to represent congestion or transmit errors -- not overworked
servers. Most importantly, transport layer drops *will* be
interpreted by the peer as congestion or transmit errors.
Inducing an incorrect response in your peer is someething that
should be avoided, even if recovery will eventually happen.

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 11 Jul 2003 13:36:41 -0700

Responding to Caitlin's two posts -

> The Data Sink ULP would still be required to authorize each reception

Generally, I agree.  It may however be the case for some ULPs (such as
iSCSI) that they may not be able to precisely predict every possible 
ULP PDU though the peers know about >95%  of the PDUs with certainty
(and can provision for them).  The ULP in this case can post a few extra
"slack" buffers, but it still is theoretically possible in some very corner cases 
that a local buffer pool may transiently run out.

John Hufferd gave an example of processing delays that could also lead to
this same scenario.

>But I disagree that it is a "benefit" to be forgiving to

Agreed, I don't think anyone is suggesting being accommodative
with bad-behaving remote peers.  I am only stating that the 
connection drop is an RNIC implementation issue, and the DDP 
spec should leave enough flexibility for implementations here while
specifying the precise semantics a DDP implementation should 
provide to its ULP (RDMAP).

>error on material that had already passed the TCP checksum.

Not sure I agree that it is "very analogous" though it has some resemblance.
In this specific case, assuming the buffers are posted by the retransmit time,
an LLP retransmit is guaranteed to recover the packet which isn't the
case in what you describe.

>not something that any ULP should ever rely upon.

No, in fact, the point in my note was that the ULP *must not* 
be privy to what's going on at the DDP/LLP interface ("in a 
ULP-transparent way" - my suggested text).

In my simplistic analysis, there are two ways that an RNIC might
deal with an inbound untagged message that doesn't have an
available posted receive buffer.

a) RNIC simply drops the connection (as is currently spec'ed in the
    DDP spec).

b) RNIC is "smarter" (or rototilled as protocol designers see it), so
    drops and recovers the packet later on via LLP means -
    all the while being transparent to the ULP.

My suggestion was that the DDP spec shouldn't mandate either
approach.

>be acceptable. 

Sounds like you're agreeing that it's reasonable to allow either
approach (a) or (b) noted above.  You however seem to want to
specify that allowance in more formal terms than I intended.
IMHO, it suffices for the DDP spec to allow implementation
flexibility without formalizing the underlying notions.  That's what
my strawman text changes attempt to do.

Regards.
--
Mallikarjun

Mallikarjun Chadalapaka

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 11 Jul 2003 16:15:55 -0500

>My suggestion was that the DDP spec shouldn't mandate either

What I meant by conditional placement being something that
the ULP never relies upon, is that the ULP should never use
the feeding of buffers independenlty of authorizing receives
as a means of flow control.

So a ULP should never authorize 100 receives, and then only
supply 50 buffers if the purpose is to *stall* the 51st
and later responses. If the ULP is supplying only 50 buffers
it is because it does not expect their to be a 51st message.

Having a nice recovery mechanism in case that estimate was
off is certainly nice, but what I think we want to avoid is
the ULP *counting* on the lack of buffers to act as a throttle.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 11 Jul 2003 16:19:33 -0500

>provide to its ULP (RDMAP).

I don't think there's a disagreement here, but my point is that
terminating the stream for non-authorized untagged messages
is feature of the protocol, not an implementation detail.

The handling of an authorized untagged message that is unfortunately
missing a buffer is what can be considered an implementation detail.
The implementation MAY terminate the connection, or it MAY do
something smarter like wait for a retransmit.

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 12 Jul 2003 00:47:39 -0400

Caitlin,

Sorry I've been slow responding to your previous message on this
topic.  I'll roll it into this message.

The executive summary is that you should be picking these bones with
the protocols, not the architecture.  I think the first paragraph of
the architecture document makes this clear.

> buffer model that must be dealt with seperately: flow control and

The architecture IS NOT defining an interface (API, or a looser Verbs
notion, or whatever else) for RDDP.  It's only modelling what the
protocols do, in terms of messages and a set of endpoint behavior
constraints.  The C notation is a terse way to express the model.
Don't read too much into it.

The RDDP architecture does not concern itself with `provisioning of
buffers' for untagged messages.  Untagged messages are, well,
messages: atomically sent and delivered blobs of data.  The buffer
abstraction in the RDDP architecture is only for tagged messages.  How
you implement receiving of untagged messages are implementation
choice.  As long as an implemention respects the relevant ordering
requirements, you can do it any way you want (shared pools, unshared
pools, zero copy, zillion copy, very large integers, light pulses, DNA
molecules, whatever).

The receive operation's (gimmie the next message or notification) role
in the architecture is to express that DDP and RDMA allow untagged
message transfer, and that there are defined sequencing rules for
buffer operations (set()s, get()s and notifications) against ULP `blob
gimmie' operations.  The protocols nail down the sequencing rules.
The architecture just says the exist (i.e. sequencing is a central
concept).

The architecture's also silent on untagged message flow control.  The
protocols are where you'd say something (and argue) about flow
control.

To repeat, the architecture document IS NOT a specification.  It's
just a model that frames what the specifications describe (as in `what
the hell are we talking about with this DDP and RDMA jazz anyway?')

If you're saying there should be an explicit model of untagged message
buffering in the architecture (forcing that concept into the protocols
as well), I'm saying no there shouldn't be.

Thanks,
  Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 12 Jul 2003 12:58:26 -0500

>Don't read too much into it.
Perhaps it does not belong in the architecture draft, but that's
where it was in the earlier drafts. The first draft required the
local ULP to authorize untagged reception by posting a buffer.
That was reduced to merely requiring the ULP to authorize untagged
reception, but being silent about where buffers came from, and
then to saying nothing about how untagged receptions are regulated.

But the end-to-end protocol needs for certain rules about buffer
usage to be explicitly stated. I suppose we can debate which draft
they belong in precisely, but that does not mean these issues can
be left totally to each implementation. There are certain required
end-to-end behaviors needed to allow ULPs to work.

I don't believe there is any debate on tagged buffers:

-- Tagged messages may only access local memory to the extent that
   is explicitly authorized by the local ULP.
-- The local interface MUST allow the local ULP to enable any
   contiguous subset of its address space through an STag.
-- The local interface MAY allow other combinations of ULP
   accessible memory to be enabled for access through an STag.
   For example, zero-based buffers or maps formed from physical
   buffer lists.
-- The local interface MUST NOT allow the ULP to enable an
   STag with memory access rights that the ULP does not itself
   already have.
-- The remote peer MUST NOT use an advertised buffer in any
   way that is not explicitly authorized by the Upper Layer
   Protocol.
-- The local ULP MUST use the local interface to enable STags
   for all access rights advertised to the remote peer. It MAY
   enable the STags for more rights, but SHOULD NOT do so if
   that places any other process or client at risk.
-- The DDP Layer MUST terminate a DDP Stream if an incoming
   DDP Segment attempts to use an STag in a way that was not
   enabled by the local ULP.
   
I believe the following rules are required for untagged buffers

-- Untagged messages may only be sent as explicitly allowed by
   the Upper Layer Protocol. The ULP MUST define some mechanism
   for granting permission to send Untagged messages.
-- Authorization for an untagged message MUST specify a maximum
   message size. Receipt of an untagged message that is larger
   MUST result in termination of the DDP Stream.
-- Authorized untagged messages SHOULD be placed into a ULP
   accessible buffer. The local interface MUST allow the ULP
   to supply its own buffer for each authorized untagged message.
   The local interface MAY alternately allow the buffers to be filled 
   from a shared resource.
-- Unauthorized untagged messages SHOULD terminate the DDP Stream.
   Failure to terminate an "over credit" DDP Stream MUST NOT imperil
   the availability of resources for any compliant DDP Stream.
-- The receiving side SHOULD have an adequate supply of buffers to
   handle all authorized untagged messages. However, this does not
   require pre-committment of one buffer for each authorized untagged
   message. Statistical sharing of buffer resources MAY be used.
   However the receiving side MUST NOT use this as a form of
   ex post facto flow control.

With these rules, iWARP has extremely powerful and flexible flow
control that is directly responsive to actual ULP needs. It is
superior to the flow control available to non-RDMA IP ULPs.

However, without these rules, an iWARP ULP is both more fragile
and more of a security risk than a conventional non-RDMA ULP.
There are several critical reasons for this:

-- Transport layer buffers no longer pace the stream

    A principal benefit of RDMA protocols is eliminating dependence
    on transport layer buffering. But it is easy to overlook the
    fact that as hated as any bottleneck is, they also have
    side-effects that may have been beneficial. The side effect
    of being forced to pass-through the narrow capacity of
    transport layer buffering was some flow  control.
    
    With an RDMA protocol there is no predicting where the next
    segment to arrive will be placed. Therefore it would be incorrect
    to hold it off because *one* of the logical destinations may have
    exhausted its capacity.
    
    The optimal interface between the LLP and DDP allows the LLP
    to pass each validated chunk/segment/whatever to the DDP as
    soon as it is validated and say, "here put this wherever it goes".
    
    Such an interface, if unchecked, would allow a remote peer to
    rapidly consume available buffers.
    
-- The LLP cannot detect violations by the remote peer

    All DDP Segments look the same to the LLP.
    
-- The ULP cannot detect violations by the remote peer

    Virtually all local interfaces that support RDMA protocols
    use a completion queue to deliver completions of untagged
    messages. Frequently these completion queues are:
    
    *   Opaque. The ULP cannot knows the contents of the queue
        except by dequeuing them serially.
    *   Shared across multiple DDP Streams.
    
    The net effect is that the ULP cannot know when an untagged
    message was received. By the time the completion is reaped,
    an indeterminant amount of time will have passed. The remote
    peer may have been granted permission during that time.
    
    Placing responsiblity for catching "cheaters" who sent one
    or two "extra" untagged messages early on the ULP will virtually
    guarantee that cheating will not be caught. If clients are
    competing for resources, for example when they are all using
    a common file server, it would be to the each of the clients
    advantage to cheat. Or at the minimum, because they are competing
    for resources, any client that has an "off by one" error will be
    far more motivated to fix it if they are sending one *too few*
    requests to the server.
    
    This could easily escalate into an arms race, where everyone
    assumes that the Server is counting on the clients cheating by
    one, and that playing by the rules would place the client at
    a competitive disadvantage versus other clients.