Subject: [rddp] Association of STag and RDMAP Stream(s) - version 3

From: Renato Recio <[email-masked]>
Date: Tue, 11 Feb 2003 13:01:54 -0600

Sorry folks,  this is a resend,  the second set of text is for the RDMA
spec.

      I believe the following text addresses the issue raised by Caitlin
regarding STag validation.

Following is text I propose to add to bullet #3 in section 10.3 of the DDP
-01 specification:

   DDP MAY choose several locally defined mechanisms for associating a DDP
   Stream and a STag. One such mechanism is to provide two association
   types: a protection domain association and an DDP Stream association.

      Under the protection domain (PD) association, a unique protection
      domain identifier (PD ID) is created and used locally to associate
      the STag with a set of DDP Streams. Under this mechanism, only DDP
      Streams that have the same PD ID as the STag are allowed to use the
      STag. For an incoming Tagged Message on an DDP Stream, if the PD ID
      associated with that DDP Stream is not the same as the PD ID
      associated with the STag targeted by the Tagged Message, then the
      Message is not placed and the DDP Layer MUST surface a local error to
      the ULP.  Note that the PD ID is locally defined, and cannot be
      directly manipulated by the Remote Peer.

      Under DDP Stream association, a given DDP Stream is identified
      locally by a unique DDP Stream identifier (ID) and that DDP Stream ID
      is associated with the STag and DDP Stream. In this case, for an
      incoming Tagged Message on an DDP Stream, if the DDP Stream ID
      associated with that DDP Stream is not the same as the DDP Stream ID
      associated with the STag targeted by the Tagged Message, then the
      Message is not placed and and the DDP Layer MUST surface a local
      error to the ULP. Note that the DDP Stream ID is locally defined, and
      cannot be directly manipulated by the Remote Peer.

Following is text I propose to add to bullet #5 in section 10.3 of the
RDMAP -01 specification:

   RDMAP MAY choose several locally defined mechanisms for associating a
   RDMAP Stream and a STag. One such mechanism is to provide two
   association types: a protection domain association and an RDMAP Stream
   association.

      Under the protection domain (PD) association, a unique protection
      domain identifier (PD ID) is created and used locally to associate
      the STag with a set of RDMAP Streams. Under this mechanism, only
      RDMAP Streams that have the same PD ID as the STag are allowed to use
      the STag.

         For an incoming RDMA Read Request Message on an RDMAP Stream, if
         the PD ID associated with that RDMAP Stream is not the same as the
         PD ID associated with the Data Source STag, the Message is not
         placed and the RDMAP Stream MUST be Terminated with an Invalid
         STag error. Finally, for a Send with Invalidate or Send with SE
         and Invalidate on an RDMAP Stream, if the PD ID associated with
         that RDMAP Stream is not the same as the PD ID associated with the
         STag that is to be Invalidated, the Message is not placed and the
         RDMAP Stream MUST be Terminated with an STag cannot be Invalidated
         error. Note that the PD ID is locally defined, and cannot be
         directly manipulated by the Remote Peer.

      Under RDMAP Stream association, a given RDMAP Stream is identified
      locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream
      ID is associated with the STag and RDMAP Stream.

         For an incoming RDMA Read Request Message on an RDMAP Stream, if
         the RDMAP Stream ID associated with that RDMAP Stream is not the
         same as the RDMAP Stream ID associated with the Data Source STag,
         then the Message is not placed and the RDMAP Stream MUST be
         Terminated with an Invalid STag error. Finally, for a Send with
         Invalidate or Send with SE and Invalidate on an RDMAP Stream, if
         the RDMAP Stream ID associated with that RDMAP Stream is not the
         same as the RDMAP Stream ID associated with the STag that is to be
         Invalidated, the Message is not placed and the RDMAP Stream MUST
         be Terminated with an STag cannot be Invalidated error. Note that
         the RDMA Stream ID is locally defined, and cannot be directly
         manipulated by the Remote Peer.

   Note:  For an incoming RDMA Write or RDMA Read Response Message on an
   RDMAP Stream, the DDP layer MUST associate the STag targeted by the RDMA
   Write or RDMA Read Response Message (respectively) to the RDMAP Stream.
   If an STag targeted by the RDMA Write or RDMA Read Response Message
   Segment is not associated with the RDMAP Stream that received the
   Message Segment,  DDP MUST surface an Invalid STag error to the RDMAP
   layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer
   surfaces an Invalid STag error.

I believe the above text resolves the problem Caitlin surfaced, folks,
please let me know if you agree.  Also,  please note,  the STag cannot be
Invalidated error is one of the two new error codes I surfaced on this
reflector.

Renato J Recio
IBM Distinguished Engineer
Member IBM Academy of Technology
IBM I/O Architecture and Technology
Tel 512-838-3685, T/L 678-3685

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Shah, Hemal" <[email-masked]>
Date: Tue, 11 Feb 2003 12:49:21 -0800

Renato,

I like the text you have for DDP and RDMAP. See my comments bounded by
<hvs>.

Hemal

-----Original Message-----
From: Renato Recio [mailto:[email-masked]] 
Sent: Tuesday, February 11, 2003 1:02 PM
To: [email-masked]
Subject: [rddp] Association of STag and RDMAP Stream(s) - version 3

Following is text I propose to add to bullet #3 in section 10.3 of the
DDP
-01 specification:

   DDP MAY choose several locally defined mechanisms for associating a
DDP
   Stream and a STag. One such mechanism is to provide two association
   types: a protection domain association and an DDP Stream association.
<hvs>
I would rephrase the above paragraph as:
There are several mechanisms for associating an STag and a DDP Stream. A
couple of reasonable mechanisms for this association are: a protection
domain association and a DDP Stream association.
<hvs>
      Under the protection domain (PD) association, a unique protection
      domain identifier (PD ID) is created and used locally to associate
      the STag with a set of DDP Streams. Under this mechanism, only DDP
      Streams that have the same PD ID as the STag are allowed to use
the
      STag. For an incoming Tagged Message on an DDP Stream, if the PD
ID
      associated with that DDP Stream is not the same as the PD ID
      associated with the STag targeted by the Tagged Message, then the
      Message is not placed and the DDP Layer MUST surface a local error
to
      the ULP.  Note that the PD ID is locally defined, and cannot be
      directly manipulated by the Remote Peer.

<hvs>
I would rephrase the above paragraph (with slight modifications) as:
Under the protection domain (PD) association, a unique protection domain
identifier (PD ID) is created and used locally to associate an STag with
a set of DDP Streams. Under this mechanism, the use of the STag is only
permitted on the DDP Streams that have the same PD ID as the STag. For
an incoming DDP Segment of a Tagged DDP Message on a DDP Stream, if the
PD ID of the DDP Stream is not the same as the PD ID of the STag
targeted by the Tagged DDP Message, then the DDP Segment is not placed
and the DDP Layer MUST surface a local error to the ULP. Note that the
PD ID is locally defined, and cannot be directly manipulated by the
Remote Peer.
<hvs>

	Under DDP Stream association, a given DDP Stream is identified
      locally by a unique DDP Stream identifier (ID) and that DDP Stream
ID
      is associated with the STag and DDP Stream. In this case, for an
      incoming Tagged Message on an DDP Stream, if the DDP Stream ID
      associated with that DDP Stream is not the same as the DDP Stream
ID
      associated with the STag targeted by the Tagged Message, then the
      Message is not placed and and the DDP Layer MUST surface a local
      error to the ULP. Note that the DDP Stream ID is locally defined,
and
      cannot be directly manipulated by the Remote Peer.

<hvs> I would rephrase the above paragraph (with slight modifications)
as:
Under DDP Stream association, a DDP Stream is identified locally by a
unique DDP Stream identifier (ID). An STag is associated with a DDP
Stream by using DDP Stream ID. In this case, for an incoming DDP Segment
of a Tagged DDP Message on a DDP Stream, if the DDP Stream ID of the DDP
Stream is not the same as the DDP Stream ID of the STag targeted by the
Tagged DDP Message, then the DDP Segment is not placed and and the DDP
Layer MUST surface a local error to the ULP. Note that the DDP Stream ID
is locally defined, and cannot be directly manipulated by the Remote
Peer.

I will also add the following paragraph to make a strong requirement
about the association of an STag and a DDP Stream:

ULP SHOULD associate an STag and a DDP Stream. DDP MUST support
protection domain association and DDP Stream association mechanisms for
associating an STag and a DDP Stream.
<hvs>

   RDMAP MAY choose several locally defined mechanisms for associating a
   RDMAP Stream and a STag. One such mechanism is to provide two
   association types: a protection domain association and an RDMAP
Stream
   association.

      Under the protection domain (PD) association, a unique protection
      domain identifier (PD ID) is created and used locally to associate
      the STag with a set of RDMAP Streams. Under this mechanism, only
      RDMAP Streams that have the same PD ID as the STag are allowed to
use
      the STag.

         For an incoming RDMA Read Request Message on an RDMAP Stream,
if
         the PD ID associated with that RDMAP Stream is not the same as
the
         PD ID associated with the Data Source STag, the Message is not
         placed and the RDMAP Stream MUST be Terminated with an Invalid
         STag error. Finally, for a Send with Invalidate or Send with SE
         and Invalidate on an RDMAP Stream, if the PD ID associated with
         that RDMAP Stream is not the same as the PD ID associated with
the
         STag that is to be Invalidated, the Message is not placed and
the
         RDMAP Stream MUST be Terminated with an STag cannot be
Invalidated
         Error. Note that the PD ID is locally defined, and cannot be
         directly manipulated by the Remote Peer.
<hvs> In case of SendInv and SendSEInv, when you check for the STag
association in RDMAP layer, the Untagged DDP Message has been already
placed and delivered to RDMAP. So, I do not think you can say that the
message is not placed when the PD ID of the STag to be invalidated and
PD ID of RDMAP Stream are not the same.
<hvs>

      Under RDMAP Stream association, a given RDMAP Stream is identified
      locally by a unique RDMAP Stream identifier (ID) and that RDMA
Stream
      ID is associated with the STag and RDMAP Stream.

         For an incoming RDMA Read Request Message on an RDMAP Stream,
if
         the RDMAP Stream ID associated with that RDMAP Stream is not
the
         same as the RDMAP Stream ID associated with the Data Source
STag,
         then the Message is not placed and the RDMAP Stream MUST be
         Terminated with an Invalid STag error. Finally, for a Send with
         Invalidate or Send with SE and Invalidate on an RDMAP Stream,
if
         the RDMAP Stream ID associated with that RDMAP Stream is not
the
         same as the RDMAP Stream ID associated with the STag that is to
be
         Invalidated, the Message is not placed and the RDMAP Stream
MUST
         be Terminated with an STag cannot be Invalidated error. Note
that
         the RDMA Stream ID is locally defined, and cannot be directly
         manipulated by the Remote Peer.
<hvs> Same comment as the paragraph for PD. In case of SendInv and
SendSEInv, when you check for the STag association in RDMAP layer, the
Untagged DDP Message has been already placed and delivered to RDMAP. So,
I do not think you can say that the message is not placed when the RDMAP
Stream ID of the STag to be invalidated and RDMAP Stream ID of RDMAP
Stream are not the same.
<hvs>

   Note:  For an incoming RDMA Write or RDMA Read Response Message on an
   RDMAP Stream, the DDP layer MUST associate the STag targeted by the
RDMA
   Write or RDMA Read Response Message (respectively) to the RDMAP
Stream.
   If an STag targeted by the RDMA Write or RDMA Read Response Message
   Segment is not associated with the RDMAP Stream that received the
   Message Segment,  DDP MUST surface an Invalid STag error to the RDMAP
   layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP
Layer
   surfaces an Invalid STag error.
<hvs> Note with slight modifications:
Note: For an incoming RDMA Write or RDMA Read Response Message on an
RDMAP Stream, the DDP layer MUST verify the association of the STag
targeted by the RDMA Write or RDMA Read Response Message (respectively)
and the underlying DDP Stream that received the Message. If an STag
targeted by the RDMA Write or RDMA Read Response Message Segment is not
associated with the underlying DDP Stream that received the Message
Segment, DDP MUST surface an Invalid STag error to the RDMAP layer. The
RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer surfaces an
Invalid STag error.
<hvs>

I believe the above text resolves the problem Caitlin surfaced, folks,
please let me know if you agree.  Also,  please note,  the STag cannot
be
Invalidated error is one of the two new error codes I surfaced on this
reflector.

Renato J Recio

---

From: [email-masked]
Date: Mon, 17 Feb 2003 15:25:42 -0500

This needs a "MUST implement this functionality or equivalent"
requirement on implementations, including a requirement for
interfaces to enable every ULP to control this behavior of
STags that it exports.  Other association models could be
allowed, but these two would need to be mandatory.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> Sent: Tuesday, February 11, 2003 2:02 PM
> To: [email-masked]
to
ID
and
use
the
the
Invalidated
Stream
be
RDMA