Subject: [rddp] verbs draft - STag validation error in 7.10.6.2

From: Caitlin Bestler <[email-masked]>
Date: Thu, 3 Jul 2003 22:39:41 -0500

Section 7.10.6.2 of the verbs draft has a mistake. In specifying the
validation that must be done for received STags it states:

>   *   The RNIC MUST check and enforce that the QP ID associated with 
>       the Memory Window is equal to the QP ID associated with the 
>       incoming remote operation that is accessing the Memory Window.

If a Memory Window has "a" QP ID associated with it then it can never
be valid on multiple QPs (DDP Streams).

But the protocol specifically states:

>   Protection Domain (PD) - A Mechanism used to associate a DDP Stream 
>       and an STag. Under this mechanism, the use of an STag is valid 
>       on a DDP Stream if the STag has the same Protection Domain 
>       Identifier (PD ID) as the DDP Stream.

An STag with the "wrong" QP ID nevertheless has the correct PD ID.
Therefore the STag is valid.

Caitlin Bestler - [email-masked] - http://asomi.com/

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Hilland, Jeff" <[email-masked]>
Date: Mon, 7 Jul 2003 10:05:44 -0500

Echoing Ellen's sentiments, this was done deliberately and with good
reason.  We encountered a design problem, looked at alternatives, and
chose this solution due to the ease of implementation, ease of use and
least perturbation to existing fabric solutions.

I would prefer that we not distract the reflector traffic to Verbs when
we haven't even nailed the wire protocols, but since I was the guilty
party to post the Verbs draft, I'll be glad to try to answer this one
and rely on our chair to determine if this conversation is premature.

When dealing with a single process which has multiple clients, each
represented by a different QP (stream), it is desirable to give each QP
only the rights to access a small amount of memory for a small amount of
time.  Memory Windows are the obvious choice.  But to get this to work
under the old VI/IB model, this means each QP must be in it's own PD (in
order to preserve security/fault domains/memory protection) and thus the
Consumer would have to register memory (Register Shared, probably) for
each PD - probably all of SGA.  In addition, each Consumer would then
have a pool of MWs for each PD.  This results in a consumption of
resources in the implementation in a fashion that is non-linear compared
to the number of connections - and thus is wasteful.  In a hardware
implementation, these may (read will) be scarce resources (at least in
lower end implementations).

After looking at several alternatives, it was determined that the amount
of time a Memory Window references a specified location is short lived.
MW usage typically follows a Bind/Use/Invalidate model.  So it was
decided that during the Bind, the MW STag would only be valid on that
one connection.  Once invalidated, it would be available for general
use.  This allowed a model where all QPs could be in one PD, all MWs
could be in one PD and you would only need to Register Memory once.
This model allows for optimum use of implementation resources and
matches most usage models.  The Verbs draft goes through this in gory
detail and suggests a method for implementing this feature.

Now, if a Consumer wants to use Memory Windows across multiple
connections, this flavor of Memory Windows is not optimal.  This was
examined as well.  And by "use", I don't mean advertise, I mean use for
RDMA Read/Write purposes.  This means that the connections share the
same access rights to advertised memory and the advertisement is long
lived (otherwise Memory Protection errors would occur).  And these
connections have a high degree of coordination of the shared MW (again,
to prevent errors that would terminate the Stream).  This model begins
to resemble the requirements of a fast-register or normal register model
and not a MW model.  So if this is a requirement, I would like to
understand why none of the alternatives are acceptable (aren't memory
registration models enough?) and would propose dialog on that usage
model to understand the requirements.

Regards,
Jeff Hilland 

-----Original Message-----
From: Deleganes, Ellen M [mailto:[email-masked]] 
Sent: Monday, July 07, 2003 9:30 AM
To: Caitlin Bestler; [email-masked]
Subject: RE: [rddp] verbs draft - STag validation error in 7.10.6.2

Making Memory Windows only valid on a single stream in the verbs was a
deliberate choice. In most programming models it makes sense to bind a
window to a stream rather than to the PD. If the application writer
wants a memory entity that can be remotely accessible by multiple
streams, they can still use Memory Regions. This decision actually makes
Memory Windows more lightweight. It is a lot quicker to do the
invalidation because you only need to check incoming traffic on a single
stream, instead of across all of the streams in a Protection Domain,
which can be rather onerous.

Ellen

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]] 
Sent: Thursday, July 03, 2003 8:40 PM
To: [email-masked]
Subject: [rddp] verbs draft - STag validation error in 7.10.6.2

>       incoming remote operation that is accessing the Memory Window.

If a Memory Window has "a" QP ID associated with it then it can never be
valid on multiple QPs (DDP Streams).