Subject: [rddp] STag Invalidation Question

From: "Williams, Jim" <[email-masked]>
Date: Fri, 7 Feb 2003 11:01:08 -0500

I am sure this issue has been discussed and debated before
although perhaps not in this forum.  I would be interested
in hearing some of the thoughts here, though.

PROPOSAL:  STags are always invalidated immediately on
           on completion of RDDP writes.  This is done
           before delivery of the data to the application.
           Therefore all tagged buffers are "write once"
           and must be re-registered before they can
           be written again.

There are some interesting advantages to this approach.

1.  It goes a long way towards resolving some of the
    issues raised in

 http://www.ietf.org/internet-drafts/draft-ietf-rddp-rdma-concerns-00.txt

    Particularly section 4.1 "Overwrites", and 4.2 "Concurrent
    Operations to the Same Memory"

2.  It eliminates the need for "remote STag invalidate"
    which strikes me as an undesirable complexity
    in the currently proposed protocol definition.

3.  It allows speculative out-of-order placement without
    the need for periodic markers or absolute alignment
    guarantees.  Specifically, if incorrect placement
    is done, it can always be caught before delivering
    to the application, and is therefore harmless.

    Incorrect placement can (based on some assumptions
    a bit too complex to describe here) be transparently
    fixed by falling back to in order placement and
    overwriting the bad data with good data.

#3 is complex enough to deserve a draft to fully explore,
however, some initial discussion would be worth while
to gage whether this path is worth going down.

No doubt there are some good arguments for the current
proposal and against mandating "one shot" tagged
buffers.  I would like to hear the justification.

 - Jim Williams

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 7 Feb 2003 11:08:23 -0600

>           and must be re-registered before they can

There are numerous reasons for wanting to re-use STags that
relate to the presumption that binding them for each
operation will be too expensive.

I fully agree that those reasons are insufficient.
Implementations SHOULD assure that binding and invalidating
an STag for each RDMA Write is a truly valid option. No ULP
should refrain from doing so for fear of performance
penalties.

Numerous algorithms developed for dual-ported memory are
applicable to RDMA and use of persistent STags.

The primary example for me remains Variable Bit Rate (VBR)
delivery. The Data Sink advertises a single large buffer
intended to accept data over an extended period of time.
Sends will be sent at timed intervals to confirm how much
data has been placed (and allowing it to be deliverd).

Because the amount of data transferred per completion is
variable, there is no way to pre-feed future STags. The
starting address for pipeline stage N begins *after* stage
N-1, and this would not have been known when a distinct STag
would have had to have been created.

Another example would be a SAN or NAS server that has been
asked to write 100 MB from a client buffer specified with an
STag and TO.

Unless it can dedicate a 100 MB buffer (unlikely), the
server will be using RDMA Reads to fetch chunks matching
half of its buffer size (perhaps 1 MB), and then writing
from those buffer to disk. (Two buffers wouuld be used at
the minimum so that one can be pre-loaded while the other
is written to disk).

These have to be distinct RDMA Reads, so that a completion
is generated for each portion. The completion would trigger
the actual write to disk.

It is outside the scope of the IETF, but I would certainly
recomend that applications SHOULD use one-shot STags, and
that support for invalidation SHOULD be efficient enough
so that no ULP would avoid one-shot STags for performance
reasons.

The applications that have need to use persistent STags have
valid reasons of the type needed to override a SHOULD. They
are advanced techniques that should not be selected by
default.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Stephen Bailey <[email-masked]>
Date: Sat, 22 Feb 2003 00:09:24 -0500

Jim,

>            and must be re-registered before they can

Others have pointed out the advantages of NOT adopting this model, but
I feel the need to further amplify their point.

Arbitrary reuse of mapped buffer is an extremely powerful mechanism
for building high performance ULPs.  RDDP should absolutely permit
arbitrary buffer reuse.

A concern is that this extreme power can result in `unsafe' ULPs (and
mappings).  That is true.  The right way to view this is that
different ULPs have different safety and performance requirements.

Applications that need utmost performance and are `closed system' can
employ less safe techiques because the environment is controlled and
the implementations are tested to a level associated with the risk.
Frequently these are applications that will never be standardized
anywhere---they come from programmers just sitting down and building
on top of supplied communication services.  Obviously, somebody
writing, say, a database application that is going to handle zillions of
dollars is going to do a little testing, and knows best about how to
get complete application safety.

On the other hand, applications that are more `open system', like
Internet data transport, with many interoperating implementations,
should be careful and avail themselves of as safe a mapping to RDDP as
possible.  Single use STags are one of the tools to do this.

This discussion recurs frequently enough that I think once the
protocol documents have settled and we've got some miles on the
implementations, we (somebody) should probably write a BCP on how to
do safe/defensive ULP mapping to RDDP, and some guidelines on when to
strive for safety in a ULP mapping/application.

Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 22 Feb 2003 12:09:24 -0600

On 2/22/03, Stephen Bailey wrote:

>
>This discussion recurs frequently enough that I think once
>the protocol documents have settled and we've got some
>miles on the implementations, we (somebody) should
>probably write a BCP on how to do safe/defensive ULP
>mapping to RDDP, and some guidelines on when to strive for
>safety in a ULP mapping/application.
>

I would suggest the following:

    The ULP MUST have a policy unambiguously establishing
    the right to update each portion of an advertised
    buffer.
    
    The ULP MUST NOT be required to disclose that policy.

Caitlin Bestler