Subject: RE: [rddp] STag validation semantics

From: "Deleganes, Ellen M" <[email-masked]>
Date: Tue, 1 Oct 2002 08:31:39 -0700

More comments within, starting with <<

Ellen

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Monday, September 30, 2002 7:51 PM
To: Deleganes, Ellen M; [email-masked]
Subject: RE: [rddp] STag validation semantics 

On 9/30/02, Deleganes, Ellen M wrote:

>>-- Exactly one connection: an STag is valid on exactly one
>>connection. This prevents all accidental over-exposure of
>>STags on multiple connections, but does so by eliminating
>>legitimate uses as well. This presents awkward coding
>>requirements for ULPs that use multiple connections within
>>a single session, such as DAFS.
>
><Ellen: I'd propose that if you want to use the remote
>invalidate feature that the STag be valid on only a single
>connection. If you want an STag that is valid across
>multiple connections, then you need to use a different
>mechanism to invalidate the STag. It doesn't seem sensible
>to use the remote invalidate when multiple connections are
>potentially using the same STag since ordering is not
>guaranteed across multiple connections.>
>

I really don't see the remote invalidate as being a specific
problem in this context. Of course I don't like the remote
invalidate -- I'd rather attach the invalidate to the Data
Sink's Receive than the Data Source's Send. If the Data Sink
already has to trust the Data Source to know which
*transaction* to invalidate the STag on, it already knows
what *stream*. Once you introduce the idea that only *some*
Sends invalidate an STag, being selective across multiple
streams is not a problem.

Of course this is all *far* simpler if the invalidation is
moved from the Send to the Recv. It also simplifies the wire
protocol.

<<Your suggestion requires a high level of coordination between the sender
and the receiver that might not always be possible. The model that I think
people were thinking about was using RDMA Write or RDMA Read after the
transfers completed a status message is sent that includes the invalidate.
The sender is the entity that knows when the transfer is complete and the
STag can be invalidated, not the receiver. The problem is if the sender
sends a status update in the middle of the transfer (perhaps just to let the
receiver know that there is still more data coming), it would not want the
STag invalidated at that point. The receiver would have no way to know
apriori when it posted the receive request whether or not to invalidate the
STag. The sender already has to know the STag in order to be able to
transfer data via RDMA transfers. 

Attempting to implement some sort of counting or bit-masked
release of an STag (where it is not totally released until
*all* claims on it are released) makes me nervous. It is
particularly tricky with the Protection Domain approach.
So if we keep invalidation on the wire, I think it does have
to be kept to a flat "this invalidates this STag" unless we
mandate  the full any to any mapping, in which case the
Stream/STag pairing can be deleted very precisely. Of course
making these tables *dynamic* makes your objection to their
complexity (next item) even more telling.

<<I agree with your statement, that implementing counting or bit-masked
release of the STag is not desireable. I also prefer the flat "this
invalidated this STag". Just like an RNIC command to invalidate an STag
would invalidate it across all accesses asynchronously, I think this should
work the same way.

>>-- Explicit: the STag is valid only on those connections
>>that the ULP has specifically enabled it for. It may
>>enable it for any set of connections. This provides
>>complete ULP flexibility. It is also a very simple
>>specification.
>
><Ellen: I really don't like this from an implementation
>viewpoint, it requires the RNIC to maintain a table of the
>valid connections for each STag. This is additional
>context and complexity for the hardware and will have a
>performance impact on each access to the memory associated
>with that STag. The other models only require a QP or PD
>match, this entails a search by connection.>
>

As an application developer I of course want this
flexibility. As an ASIC designer I even more emphatically
agree with you. As an Architect I'd tell the application
developer to take her lumps and work around it. A Protection
Domain like limitation on the scope of STag validity is
essential to efficient hardware.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 1 Oct 2002 14:55:33 -0500

On 10/1/02, Deleganes, Ellen M wrote:

>>moved from the Send to the Recv. It also simplifies the
>>wire protocol.
>
><<Your suggestion requires a high level of coordination
>between the sender and the receiver that might not always
>be possible. The model that I think people were thinking
>about was using RDMA Write or RDMA Read after the
>The sender is the entity that knows when the transfer is
>complete and the STag can be invalidated, not the
>receiver. The problem is if the sender sends a status
>update in the middle of the transfer (perhaps just to let
>the receiver know that there is still more data coming),
>it would not want the STag invalidated at that point. The
>receiver would have no way to know apriori when it posted
>the receive request whether or not to invalidate the STag.
>The sender already has to know the STag in order to be
>able to transfer data via RDMA transfers. 
>

It's very simple co-ordination. It merely requires that the
Data Sink (typically the client) be in the state of
expecting a response to its request.

But as simple and logical as priming the Recv queue with
buffers to match the state of the protocol is, it is not the
only strategy. A ULP could indeed use a standard buffer size
for all untagged messages and have no expectation as to the
next message.

Personally, I think that such a strategy is a lot more
complicated and only justified for true peer-to-peer
applications. Even then I might be tempted to set up two
streams, each maintaining the tradtional client/server role
in each direction -- but whether or not I would use it, it
is indeed a valid strategy. And you are correct, for that
strategy the invalidate has to be attached to the Send
operation rather than the receive.

Caitlin Bestler

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 4 Oct 2002 17:52:18 +0300

Ellen,

I would like also to point out that protocols that allow STags and 
invalidation on several connections will have hard time avoid a myriad of 
race conditions.

Julo

"Deleganes, Ellen M" <[email-masked]>
Sent by: [email-masked]
01/10/02 17:31
 
        To:     "'Caitlin Bestler'" <[email-masked]>, [email-masked]
        cc: 
        Subject:        RE: [rddp] STag validation semantics

Ellen

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Monday, September 30, 2002 7:51 PM
To: Deleganes, Ellen M; [email-masked]
Subject: RE: [rddp] STag validation semantics 

sends a status update in the middle of the transfer (perhaps just to let 
the
apriori when it posted the receive request whether or not to invalidate 
the

<<I agree with your statement, that implementing counting or bit-masked
release of the STag is not desireable. I also prefer the flat "this
invalidated this STag". Just like an RNIC command to invalidate an STag
would invalidate it across all accesses asynchronously, I think this 
should
work the same way.

Caitlin Bestler

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 4 Oct 2002 10:41:33 -0500

On 10/4/02, Julian Satran wrote:

>Ellen,
>
>I would like also to point out that protocols that allow
>STags and invalidation on several connections will have
>hard time avoid a myriad of race conditions.
>
Invalidation would be requested at the completion of a
transaction or transaction set. Even if there are multiple
connections in a session, each will tyically serve a
purpose. Therefore knowing when to invalidate and on which
connection will not typically be a problem for most ULPs.

For example: DAFS allows multiple connections. Even if this
feature was used (it actually would not be needed with
RDMAP) the only SENDs that invalidated as STag would be sent
by the client on the primary connection after all transfers
on the RDMA Read channel had completed.

My initial inclination was to keep STag invalidation purely
local. Ellen pointed out the scenario where the next SEND
might be an exception rather than a response. In that case
it would truly be up to the sender to indicate which message
was the response.

There are two ways the problems of multiple paths to
invalidation could be solved:

- Do not allow it. Remote invalidation is only defined
  for STags in use on a single connection. One step down
  from there, allow it but add a great big UNDEFINED
  warning and state the ULPs SHOULD NOT do this, but
  that NICs are not responsible for stopping them.
  
- Define an 'exception' message queue, such that the
  message that will arrive on the main untagged queue
  can be made predictable once again. It's a nice feature,
  but I do not think its worth the complexity.
  
If an application finds automatic invalidation to be
problematic, it can simply revert to local ULP invalidation.
That option will always be there.

Caitlin Bestler