Subject: Re: [rddp] Association of STag and RDMAP Stream(s)

From: Renato Recio <[email-masked]>
Date: Tue, 11 Feb 2003 12:16:50 -0600

Caitlin,

   My reponses bounded by <rr>.

Renato J Recio
IBM Distinguished Engineer
Member IBM Academy of Technology
IBM I/O Architecture and Technology
Tel 512-838-3685, T/L 678-3685

                      Caitlin Bestler                                                                                   
                      <[email-masked]>         To:       Renato Recio/Austin/IBM@IBMUS, [email-masked]                   
                      Sent by:                 cc:                                                                      
                      [email-masked]        Subject:  Re: [rddp] Association of STag and RDMAP Stream(s)             
                      rg                                                                                                

                      02/11/2003 11:09                                                                                  
                      AM                                                                                                

On 2/11/03, Renato Recio wrote:

>
>
>
>
>Folks,
> A problem was raised on this reflector regarding the need
> for a little more text describing the mechanisms the
> RDMAP > layer may use to associate an RDMAP Stream and an
> STag.
>
> Following is text I propose to add to bullet #5 in
> section 10.3 of the -01 specification:
>
>

Two points:

First, I believe that validation of STags is a DDP issue,
not RDMAP. So the extra wording should go in *that* draft.

<rr> In my view, validation of STags is performed by DDP for RDMA Writes
and RDMA Read Responses. It is performed by RDMA for RDMA Read Requests,
Send with Invalidate, and Send with SE and Invalidate.

I think the RDMA text should span all messages that perform STag
validation. However, I agree the text should span the two specifications.
I'll offer the suggested text shortly.<rr>

STag Invalidation may be performed by RDMAP, but validation
must be performed *before* placement, and therefore must be
handled by the DDP layer.

<rr> Agreed for RDMA Writes and RDMA Read Responses.  However, DDP is
unaware that Untagged Messages carry an STag and cannot do STag validation
for RDMA Read Requests, Send with Invalidate, and Send with SE and
Invalidate.  For these messages RDMA must perform STag validation. <rr>

Second, I believe it should be made clear that this does not
imply a standard scope for an STag. The naming domain
obviously must be at least as large as the domain over which
it is valid, but it could be only that size, the scope of
the IP address, the subnet or whatever.

<rr> The additional text does not mention the scope of an STag and I agree
it should not. Though the STag scope is dependent on the mechanism used to
associate the STag to an RDMAP Stream and two such mechanisms are described
in my e-mail, the RDMAP layer is free to choose whether to support one,
both, or neither of the alternatives I described. <rr>

If you use an STag somewhere else, you have no guarantee
that you will not accidentally pick a valid value.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Thu, 13 Feb 2003 11:56:01 -0700

Renato,

This part of the text isn't quite right:
   For an incoming RDMA Read Request
   Message on an RDMAP Stream, if the PD ID associated with that RDMAP
   Stream is not the same as the PD ID associated with the Data Source
   STag, the Message is not placed and the RDMAP Stream MUST be Terminated.
         ^^^^^^^^^^^^^^^^^^^^^^^^^
   Finally, for a Send with Invalidate or Send with SE and Invalidate on an
   RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the
   same as the PD ID associated with the STag that is to be Invalidated,
   the Message is not placed and the RDMAP Stream MUST be Terminated.
   ^^^^^^^^^^^^^^^^^^^^^^^^^

For RDMA Read Request, Send with Invalidate and Send with SE and Invalidate, DDP is doing the placement before it delivers the message to RDMAP and RDMAP is doing the STag validity check. The messages will be placed. The underlined text for RDMA Read Request should be replaced with "no RDMA Read Response is sent". The underlined text for Send with Invalidate and Send with SE and Invalidate should be "the Message is not delivered to the ULP". The corrections also apply to the Stream association paragraph.

Regards,
Pat

-----Original Message-----
From: Renato Recio [mailto:[email-masked]]
Sent: Tuesday, February 11, 2003 8:06 AM
To: [email-masked]
Subject: [rddp] Association of STag and RDMAP Stream(s)

Folks,
      A problem was raised on this reflector regarding the need for a
little more text describing the mechanisms the RDMAP layer may use to
associate an RDMAP Stream and an STag.

      Following is text I propose to add to bullet #5 in section 10.3 of
the -01 specification:

RDMAP MAY choose several locally defined mechanisms for associating a RDMAP
Stream and a STag. One such mechanism is to provide two association types:
a protection domain association and an RDMAP Stream association.

   Under the protection domain (PD) association, a unique protection domain
   identifier (PD ID) is created and used locally to associate the STag
   with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams
   that have the same PD ID as the STag are allowed to use the STag. For an
   incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if
   the PD ID associated with that RDMAP Stream is not the same as the PD ID
   associated with the STag targeted by the RDMA Write or RDMA Read
   Response Message (respectively), then the Message is not placed and the
   RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request
   Message on an RDMAP Stream, if the PD ID associated with that RDMAP
   Stream is not the same as the PD ID associated with the Data Source
   STag, the Message is not placed and the RDMAP Stream MUST be Terminated.
   Finally, for a Send with Invalidate or Send with SE and Invalidate on an
   RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the
   same as the PD ID associated with the STag that is to be Invalidated,
   the Message is not placed and the RDMAP Stream MUST be Terminated. Note
   that the PD ID is locally defined, and cannot be directly manipulated by
   the Remote Peer.

   Under RDMAP Stream association, a given RDMAP Stream is identified
   locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID
   is associated with the STag and RDMAP Stream. In this case, for an
   incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if
   the RDMAP Stream ID associated with that RDMAP Stream is not the same as
   the RDMAP Stream ID associated with the STag targeted by the RDMA Write
   or RDMA Read Response Message (respectively), then the Message is not
   placed and the RDMAP Stream MUST be Terminated. Similarly, for an
   incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP
   Stream ID associated with that RDMAP Stream is not the same as the RDMAP
   Stream ID associated with the Data Source STag, then the Message is not
   placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with
   Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the
   RDMAP Stream ID associated with that RDMAP Stream is not the same as the
   RDMAP Stream ID associated with the STag that is to be Invalidated, the
   Message is not placed and the RDMAP Stream MUST be Terminated. Note that
   the RDMA Stream ID is locally defined, and cannot be directly
   manipulated by the Remote Peer.

I believe the above text resolves the problem, folks, please let me know if
you agree.

Thanks,

Renato J Recio

---

From: "Culley, Paul" <[email-masked]>
Date: Thu, 13 Feb 2003 13:27:23 -0600

For the Send with invalidate cases, I disagree with Pat, I think the
correct action is to inform the consumer that the invalidate failed, and
why, (Invalidate Stag PD invalid), and to do this when the message is
delivered to the consumer.  Assuming there are not other errors, the
Send message is placeable and deliverable.

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, February 13, 2003 12:56 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Association of STag and RDMAP Stream(s)

Renato,

This part of the text isn't quite right:
   For an incoming RDMA Read Request
   Message on an RDMAP Stream, if the PD ID associated with that RDMAP
   Stream is not the same as the PD ID associated with the Data Source
   STag, the Message is not placed and the RDMAP Stream MUST be
Terminated.
         ^^^^^^^^^^^^^^^^^^^^^^^^^
   Finally, for a Send with Invalidate or Send with SE and Invalidate on
an
   RDMAP Stream, if the PD ID associated with that RDMAP Stream is not
the
   same as the PD ID associated with the STag that is to be Invalidated,
   the Message is not placed and the RDMAP Stream MUST be Terminated.
   ^^^^^^^^^^^^^^^^^^^^^^^^^

For RDMA Read Request, Send with Invalidate and Send with SE and
Invalidate, DDP is doing the placement before it delivers the message to
RDMAP and RDMAP is doing the STag validity check. The messages will be
placed. The underlined text for RDMA Read Request should be replaced
with "no RDMA Read Response is sent". The underlined text for Send with
Invalidate and Send with SE and Invalidate should be "the Message is not
delivered to the ULP". The corrections also apply to the Stream
association paragraph.

Regards,
Pat

-----Original Message-----
From: Renato Recio [mailto:[email-masked]]
Sent: Tuesday, February 11, 2003 8:06 AM
To: [email-masked]
Subject: [rddp] Association of STag and RDMAP Stream(s)

Folks,

RDMAP MAY choose several locally defined mechanisms for associating a
RDMAP Stream and a STag. One such mechanism is to provide two
association types: a protection domain association and an RDMAP Stream
association.

   Under the protection domain (PD) association, a unique protection
domain
   identifier (PD ID) is created and used locally to associate the STag
   with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams
   that have the same PD ID as the STag are allowed to use the STag. For
an
   incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream,
if
   the PD ID associated with that RDMAP Stream is not the same as the PD
ID
   associated with the STag targeted by the RDMA Write or RDMA Read
   Response Message (respectively), then the Message is not placed and
the
   RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request
   Message on an RDMAP Stream, if the PD ID associated with that RDMAP
   Stream is not the same as the PD ID associated with the Data Source
   STag, the Message is not placed and the RDMAP Stream MUST be
Terminated.
   Finally, for a Send with Invalidate or Send with SE and Invalidate on
an
   RDMAP Stream, if the PD ID associated with that RDMAP Stream is not
the
   same as the PD ID associated with the STag that is to be Invalidated,
   the Message is not placed and the RDMAP Stream MUST be Terminated.
Note
   that the PD ID is locally defined, and cannot be directly manipulated
by
   the Remote Peer.

   Under RDMAP Stream association, a given RDMAP Stream is identified
   locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream
ID
   is associated with the STag and RDMAP Stream. In this case, for an
   incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream,
if
   the RDMAP Stream ID associated with that RDMAP Stream is not the same
as
   the RDMAP Stream ID associated with the STag targeted by the RDMA
Write
   or RDMA Read Response Message (respectively), then the Message is not
   placed and the RDMAP Stream MUST be Terminated. Similarly, for an
   incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP
   Stream ID associated with that RDMAP Stream is not the same as the
RDMAP
   Stream ID associated with the Data Source STag, then the Message is
not
   placed and the RDMAP Stream MUST be Terminated. Finally, for a Send
with
   Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the
   RDMAP Stream ID associated with that RDMAP Stream is not the same as
the
   RDMAP Stream ID associated with the STag that is to be Invalidated,
the
   Message is not placed and the RDMAP Stream MUST be Terminated. Note
that
   the RDMA Stream ID is locally defined, and cannot be directly
   manipulated by the Remote Peer.

I believe the above text resolves the problem, folks, please let me know
if you agree.

Thanks,

Renato J Recio

---

From: [email-masked]
Date: Thu, 13 Feb 2003 12:54:42 -0700

Paul, 

I considered that possibility and I don't feel strongly about it. However, what I proposed is consistant with what is done for other error cases. The stream is being terminated. Do you deliver the last errored message or not. In other error cases where we terminate, I don't think we specify delivering the message though we do pass the headers in the terminate message and I assume that an implementation would make at least the header available locally for debug purposes. 

One knows that either something is broken or someone is maliciously trying to invalidate an STag they shouldn't access. Other then debug, one wouldn't want the ULP to act on the message which might have been malicious. One could deliver the message with an error indication and rely on the ULP to not act on it. It isn't clear that that accomplishes a useful purpose.

Regards,
Pat

-----Original Message-----
From: Culley, Paul [mailto:[email-masked]]
Sent: Thursday, February 13, 2003 11:27 AM
To: [email-masked]
Subject: RE: [rddp] Association of STag and RDMAP Stream(s)

Paul R. Culley
HP Senior Fellow
281-514-5543

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, February 13, 2003 12:56 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Association of STag and RDMAP Stream(s)

Renato,

Regards,
Pat

-----Original Message-----
From: Renato Recio [mailto:[email-masked]]
Sent: Tuesday, February 11, 2003 8:06 AM
To: [email-masked]
Subject: [rddp] Association of STag and RDMAP Stream(s)

Folks,

Thanks,

Renato J Recio