Subject: [rddp] RDMAP: Ordering and Completions (7.5) - rule 12

From: Caitlin Bestler <[email-masked]>
Date: Tue, 5 Nov 2002 12:48:49 -0600

The wording of rule 12 was slightly cleaned up to the
following:

> Note the ULP is assured that data written can be read
> back. If an RDMA Read Request is issued by the local
> peer, targeting the same ULP Buffer as a preceding Send
> or RDMA Write (in the same direction as the RDMA Read
> Request), the remote peer will send back the data written
> by the Send or RDMA Write. Thus assuring that subsequent
> local or remote accesses to the ULP Buffer contain the
> data written by the Send or RDMA Write. Note: this
> statement requires that the buffers are not arranged such
> that the data is over-written either by the incoming
> data, or other applications.

Is the intent of this clause simply that an RDMA Read
Response MUST NOT contain any content older than the prior
update to the same ULP buffer within the same stream?
However there are no guarantees made about other sources of
updates, whether local or via a different stream. In
particular the fact that an update from another source
completed "earlier" DOES NOT guarantee that said later
content will be reflected in the RDMA Read response. (Any
attempt to define when it will would essentially be trying
to write a standard on how multi-processor caching must
work, and I don't think any of us have the time to do that).

Further, when is the implementation required to catch that
it is expected to deliver from its unflushed cache (and/or
fence the read behind flushing the cache) rather than the
still un-updated  host memory?

- Only when the buffer references are *identicial*? That is,
same STag, Address and length?

- The above, or a subset thereof? This is slightly more
complex checking, but probably should be legitimate. It is
not obvioius that reading back a *portion* of a buffer is
less safe than reading back the *entire* buffer.

- When the STag is the same? (This could be tricky to
implement because it would require mixing the source, or
fencing the read behind the cache flush to host memory).

- At any time? (Not feasible without fencing of reads).

I could see either of the first two, but if it is *exact*
match only then I believe a more explicit warning is
required.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Wed, 6 Nov 2002 10:52:27 +0200

This rule starts with:

"RDMA Read Request Messages, including zero-length RDMA Read
Requests, MUST NOT start processing at the Remote Peer until
they have been Delivered to RDMAP by DDP."

DDP delivery means that all preceeding DDP messages has been completed, or
PLACEMENT of all DDP segments of those messages was finished. Assuming that
this is a case, and Read operation is performed after the PLACEMENT is
completed, the Read Response should carry data updated by all preceeding
Sends and Writes (this does not protect from the placement of the following
operations).

The first rule in the same section states that:

"RDMAP does not provide ordering among Messages on different
RDMAP Streams."

So it's ULP responsibility to take take care of
different-streams-accessing-the-same-buffers case.

Caching and fault-zone definition is really implementation issue ...

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           05/11/02 08:48 PM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       [email-masked]                                                                                                 |
  |       cc:                                                                                                                     |
  |       Subject:  [rddp] RDMAP: Ordering and Completions (7.5) - rule 12                                                        |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

Caitlin Bestler

---

From: "Culley, Paul" <[email-masked]>
Date: Wed, 6 Nov 2002 08:14:53 -0600

embedded...

Paul Culley
HP Sr. Fellow
281-514-5543

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Tuesday, November 05, 2002 12:49 PM
To: [email-masked]
Subject: [rddp] RDMAP: Ordering and Completions (7.5) - rule 12

Is the intent of this clause simply that an RDMA Read
Response MUST NOT contain any content older than the prior
update to the same ULP buffer within the same stream?
<prc> Yes, although that seems a hard way to say it.  All prior Writes and Sends from the same stream are DONE, they can be read by RDMA Read on this stream, or by any other means from any other source that is allowed to read that data. <prc>

<prc> Agreed, we are only talking about this stream...

- At any time? (Not feasible without fencing of reads).
<prc> Yes.  The intent is that an RDMA Read at the data source MUST flush the prior writes to a fully coherent memory domain before it loads data and returns it.  So an RDMA Read can be used as a write/write fence...
<prc>

Caitlin Bestler