Subject: [storm] WG Last Call - iSER - comments

From: <[email-masked]>
Date: Fri, 2 Dec 2011 19:00:39 -0500

This email is sent with WG chair hat off - all of these comments are for further discussion,
but they do need to be dealt with.

In addition to my previous comment on references:

> The iSER draft currently references RFC 3720 for iSCSI - that reference will need to be
> updated to at least the new consolidated iSCSI draft, and a reference to the iSCSI new
> features (SAM) draft should probably be added.

I've now done a relatively thorough review of draft-ietf-storm-iser-05.  It looks very good
overall, but I did find a number of additional things that need attention (items 5,6 and 8
are tagged as Major items):

1) Nit - In section 1.1's definition of connection, change "logical circuit" to "logical
bi-directional communication channel".

2) The SAM-2 reference needs to be updated to the version of SAM-5 that the iSCSI SAM
new features draft uses, and that new features draft does need to be added as a normative
reference.

3) There's a discussion of markers in Section 2.1.  I'd prefer that it be removed, but I
could live with it remaining, although it would have to informatively reference RFC 3720,
not the new consolidated iSCSI draft.

4) In section 2.3, items 1 and 2 are inconsistent, as they use "session" and "connection"
for essentially the same concept.  I suggest changing "session" to "connection in item 1.

5) Major: There's something wrong with the discussion of how to send unsolicited data.
Item 4 in section 2.3 requires use of tagged buffers (RDMA), but the second paragraph in
Section 2.6 requires use of Send (untagged buffers, not RDMA), the new key in 6.9 appears
to allow unsolicited data in a tagged buffer (RDMA), but the next to last paragraph in
7.3.4 requires use of Send (untagged buffers, not RDMA).

6) Major: I think something is missing in Section 5 to explains how to conduct iSCSI
negotiation when the connections start up in iSER-assisted mode.  I assume that this is
done via Send messages and the resource allocation referred to is the resources for RDMA,
but that does need to be explained.

7) The first paragraph of Section 8.2 describes what happens when iSERHelloRequired is
negotiated to "Yes" - add a few sentence to explain what happens when it's negotiated
to "No", which is the typical case for implementations.  A similar problem occurs in
10.1.3.4 - that one's easily handled by saying that these errors cannot occur in the "No"
case.  Please check for all other dependencies on the negotiated value of
iSERHelloRequired, and make sure that both the "Yes" and "No" cases are covered.

8) Major: The IANA Considerations section is empty.  That is wrong - the new keys defined
in sections 6.8 - 6.10 need to be registered in the iSCSI Login/Text Keys registry at: 	http://www.iana.org/assignments/iscsi-parameters
IANA also should be requested to update the registrations of the other 4 iSER keys in that
registry to reference the RFC number of this draft when it is published as an RFC.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Michael Ko <[email-masked]>
Date: Sun, 04 Dec 2011 12:31:55 -0800

David,

Thanks for the review.  My comments  are inline.

Mike
----- Original Message ----- 
From: [email-masked]
To: [email-masked]
Sent: Friday, December 02, 2011 4:00 PM
Subject: [storm] WG Last Call - iSER - comments

This email is sent with WG chair hat off - all of these comments are for 
further discussion,
but they do need to be dealt with.

> features (SAM) draft should probably be added.

[mk] Done.

I've now done a relatively thorough review of draft-ietf-storm-iser-05.  It 
looks very good
overall, but I did find a number of additional things that need attention 
(items 5,6 and 8
are tagged as Major items):

1) Nit - In section 1.1's definition of connection, change "logical circuit" 
to "logical
bi-directional communication channel".

[mk] Done

2) The SAM-2 reference needs to be updated to the version of SAM-5 that the 
iSCSI SAM
new features draft uses, and that new features draft does need to be added 
as a normative
reference.

[mk] Done

3) There's a discussion of markers in Section 2.1.  I'd prefer that it be 
removed, but I
could live with it remaining, although it would have to informatively 
reference RFC 3720,
not the new consolidated iSCSI draft.

[mk] I have removed all references to markers.

4) In section 2.3, items 1 and 2 are inconsistent, as they use "session" and 
"connection"
for essentially the same concept.  I suggest changing "session" to 
"connection in item 1.

[mk] iSER-assisted mode is negotiated using the RDMAExtensions key in the 
leading connection, as stated in sections 5.1 and 6.3, to ensure that "an 
entire iSCSI session can only operate in one mode".  Hence the choice of the 
words "session" and "connection" in items 1 and 2 are correct.  I have added 
"in the leading connection" before "for each session" in the first sentence 
in item 1.

5) Major: There's something wrong with the discussion of how to send 
unsolicited data.
Item 4 in section 2.3 requires use of tagged buffers (RDMA), but the second 
paragraph in
Section 2.6 requires use of Send (untagged buffers, not RDMA), the new key 
in 6.9 appears
to allow unsolicited data in a tagged buffer (RDMA), but the next to last 
paragraph in
7.3.4 requires use of Send (untagged buffers, not RDMA).

[mk] Item 4 states that the control and all data information associated with 
the iSCSI data-type PDUs are handled in iSER using RDMA Write, RDMA Read 
Request and RDMA Read Response Messages.  The reader is referred to section 
7.1 for the meaning of "iSCSI data-type PDUs", but item 4 itself does not 
mention tagged buffers.  I have reworded item 4 for clarity.  The new key in 
6.9 is intended to resolve the issue of how the I/O buffer is used.  In 
iSCSI (RFC3270), the I/O buffer is used to contain all the data associated 
with the SCSI operation.  Some of this data can be transferred in an 
unsolicited fashion (using Send Messages), while the rest is transferred 
using RDMA.  This new key restricts of the use of this buffer for solicited 
data only, as stated in 6.9.  2.6 and 7.3.4 are correct as stated.

6) Major: I think something is missing in Section 5 to explains how to 
conduct iSCSI
negotiation when the connections start up in iSER-assisted mode.  I assume 
that this is
done via Send messages and the resource allocation referred to is the 
resources for RDMA,
but that does need to be explained.

[mk] In this version, iSER is changed to remove the requirement that the 
connection transitions from TCP mode to RDMA mode.  It does not require that 
login negotiations be done using Send Messages.

7) The first paragraph of Section 8.2 describes what happens when 
iSERHelloRequired is
negotiated to "Yes" - add a few sentence to explain what happens when it's 
negotiated
to "No", which is the typical case for implementations.  A similar problem 
occurs in
10.1.3.4 - that one's easily handled by saying that these errors cannot 
occur in the "No"
case.  Please check for all other dependencies on the negotiated value of
iSERHelloRequired, and make sure that both the "Yes" and "No" cases are 
covered.

[mk] Done

8) Major: The IANA Considerations section is empty.  That is wrong - the new 
keys defined
in sections 6.8 - 6.10 need to be registered in the iSCSI Login/Text Keys 
registry at: http://www.iana.org/assignments/iscsi-parameters
IANA also should be requested to update the registrations of the other 4 
iSER keys in that
registry to reference the RFC number of this draft when it is published as 
an RFC.

[mk] Done.

Thanks,
--David

_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: <[email-masked]>
Date: Tue, 6 Dec 2011 00:26:25 -0500

Hi Mike,

Thanks for the quick response - your changes for items 1-4 and 8 in the -06 version look fine to me.

For item 5, I see where I got confused - the definitions section (1.1) and Section 7.1 both define the SCSI Data-out PDU to not be a data-type PDU.  That's somewhat counter-intuitive, but can be handled with some clarifying text.  I suggest the following clarifying change to item 4 in section 2.3 (my new text probably isn't 100% correct, but should be close):

OLD
   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
      are used for carrying control and all data information associated
      with the iSCSI data-type PDUs.  See section 7.1 for more details
      on iSCSI data-type PDUs.
NEW
   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
      are used for carrying control and all data information associated
      with the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs).
iSCSI SCSI Data-out PDUs do not use RDMA transfer mechanisms because
the RDMA Read operation that replaces the R2T PDU obviates the use
of SCSI Data-out PDUs for solicited Data, and SCSI Data-out
PDUs for unsolicited data are transferred by Send Messages instead of
using RDMA.  See section 7.1 for more details on iSCSI data-type PDUs.
END

For item 6 - if the connection starts in RDMA mode, how is login negotiation conducted?  I think the answer is Send Messages.  In any case, a sentence or two that answers this question in the first paragraph of Section 5.1 is what I'm looking for, as it's easy to mis-read that paragraph as requiring a start in normal TCP mode, not RDMA mode.

For item 7, the new text says:

      If iSERHelloRequired is negotiated to "No", then the maximum number
      of RDMA Read operations that can be active is negotiated via other
      means.

I suggest changing "other means" to "other means outside the scope of this document" and providing an example of "other means".

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Sunday, December 04, 2011 3:32 PM
To: Black, David; [email-masked]
Subject: Re: [storm] WG Last Call - iSER - comments

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]>
Sent: Friday, December 02, 2011 4:00 PM
Subject: [storm] WG Last Call - iSER - comments

[mk] Done.

[mk] Done

[mk] Done

[mk] iSER-assisted mode is negotiated using the RDMAExtensions key in the leading connection, as stated in sections 5.1 and 6.3, to ensure that "an entire iSCSI session can only operate in one mode".  Hence the choice of the words "session" and "connection" in items 1 and 2 are correct.  I have added "in the leading connection" before "for each session" in the first sentence in item 1.

[mk] Item 4 states that the control and all data information associated with the iSCSI data-type PDUs are handled in iSER using RDMA Write, RDMA Read Request and RDMA Read Response Messages.  The reader is referred to section 7.1 for the meaning of "iSCSI data-type PDUs", but item 4 itself does not mention tagged buffers.  I have reworded item 4 for clarity.  The new key in 6.9 is intended to resolve the issue of how the I/O buffer is used.  In iSCSI (RFC3270), the I/O buffer is used to contain all the data associated with the SCSI operation.  Some of this data can be transferred in an unsolicited fashion (using Send Messages), while the rest is transferred using RDMA.  This new key restricts of the use of this buffer for solicited data only, as stated in 6.9.  2.6 and 7.3.4 are correct as stated.

[mk] In this version, iSER is changed to remove the requirement that the connection transitions from TCP mode to RDMA mode.  It does not require that login negotiations be done using Send Messages.

[mk] Done

[mk] Done.

Thanks,
--David
[email-masked]<mailto:[email-masked]> Mobile: +1 (978) 394-7754

_______________________________________________
storm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storm

---

From: Michael Ko <[email-masked]>
Date: Thu, 08 Dec 2011 17:21:40 -0800

David,

My comments are inline.  If this is acceptable to you, I can upload the 
revised draft.

Mike
----- Original Message ----- 
From: [email-masked]
To: [email-masked] ; [email-masked]
Sent: Monday, December 05, 2011 11:26 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Hi Mike,

Thanks for the quick response - your changes for items 1-4 and 8 in the -06 
version look fine to me.

For item 5, I see where I got confused - the definitions section (1.1) and 
Section 7.1 both define the SCSI Data-out PDU to not be a data-type PDU. 
That's somewhat counter-intuitive, but can be handled with some clarifying 
text.  I suggest the following clarifying change to item 4 in section 2.3 
(my new text probably isn't 100% correct, but should be close):

OLD

      on iSCSI data-type PDUs.

NEW

END

[mk] I have added the following at the end of item 4: "As described in 
section 7.3.4, R2Ts are transformed by the iSER layer at the target into 
RDMA Read operations.  Therefore solicited SCSI write data are sent using 
RDMA Read Response Messages instead of the SCSI Data-out PDUs.  For 
unsolicited SCSI Write data, the iSER Layer at the initiator uses Send 
Messages to send the SCSI Data-out PDUs."

For item 6 - if the connection starts in RDMA mode, how is login negotiation 
conducted?  I think the answer is Send Messages.  In any case, a sentence or 
two that answers this question in the first paragraph of Section 5.1 is what 
I'm looking for, as it's easy to mis-read that paragraph as requiring a 
start in normal TCP mode, not RDMA mode.

[mk] I have added the following paragraph after the first paragraph in 
section 5: "For a transport layer that operates in byte stream mode such as 
TCP, the RCaP implementation supports the enabling of the RDMA mode after 
Connection establishment and the exchange of Login parameters in byte stream 
mode.  For a transport layer that provides message delivery capability such 
as [IB], the RCaP implementation supports the use of the messaging 
capability by the iSCSI Layer directly for the Login phase after connection 
establishment before enabling iSER-assisted mode."

      means.

I suggest changing "other means" to "other means outside the scope of this 
document" and providing an example of "other means".

[mk] I have added the following after "other means": "outside the scope of 
this document.  For example, in InfiniBand, iSER connection setup uses 
InfiniBand CM MADs, with additional iSER information exchanged in the 
private data."

Thanks,
--David

David,

Mike

----- Original Message ----- 

To: [email-masked]

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: <[email-masked]>
Date: Fri, 9 Dec 2011 17:52:38 -0500

Mike,

[still with WG chair hat off]

Your proposed text changes for items 6 and 7 are fine with me.

For item 5, I think it's important that Section 2.3 item 4 make it clear that for iSER, the SCSI Data-out PDU is *not* a data-type PDU.  My proposed text was a bit weak on that point, but I didn't see it at all in your proposed text.

Here's another attempt at new text (combining text from both of us and editing further):

   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
      are used for carrying control and all data information associated
      with the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs).
As described in section 7.3.4, R2Ts are transformed by the iSER layer
at the target into RDMA Read operations. Therefore, iSER sends
solicited SCSI write data using RDMA Read Response Messages instead
of SCSI Data-out PDUs.  iSER uses SCSI Data-out PDUs only for unsolicited
SCSI Write data, which is sent via Send Messages instead of RDMA.  Because
RDMA is not used, the SCSI Data-out PDUs for unsolicited data are not
considered to be iSCSI data-type PDUs for iSER. See section 7.1 for more
discussion of iSCSI data-type PDUs.

Unfortunately, this is still somewhat subtle and potentially confusing (the Data-out PDU is not a data-type PDU, rather the Data-out PDU is a control-type PDU ... huh??); the alternative is probably a serious rewrite to replace the terms "data-type PDU" and "control-type PDU" with terms that don't use "data" and "control", e.g., "RDMA-transfer PDU" and "message-transfer PDU".

What do you think?

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Thursday, December 08, 2011 8:22 PM
To: Black, David; [email-masked]
Subject: Re: [storm] WG Last Call - iSER - comments

David,

My comments are inline.  If this is acceptable to you, I can upload the revised draft.

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, December 05, 2011 11:26 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Hi Mike,

OLD

[mk] I have added the following at the end of item 4: "As described in section 7.3.4, R2Ts are transformed by the iSER layer at the target into RDMA Read operations.  Therefore solicited SCSI write data are sent using RDMA Read Response Messages instead of the SCSI Data-out PDUs.  For unsolicited SCSI Write data, the iSER Layer at the initiator uses Send Messages to send the SCSI Data-out PDUs."

[mk] I have added the following paragraph after the first paragraph in section 5: "For a transport layer that operates in byte stream mode such as TCP, the RCaP implementation supports the enabling of the RDMA mode after Connection establishment and the exchange of Login parameters in byte stream mode.  For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode."

[mk] I have added the following after "other means": "outside the scope of this document.  For example, in InfiniBand, iSER connection setup uses InfiniBand CM MADs, with additional iSER information exchanged in the private data."

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]>
Sent: Friday, December 02, 2011 4:00 PM
Subject: [storm] WG Last Call - iSER - comments

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: Michael Ko <[email-masked]>
Date: Fri, 09 Dec 2011 16:24:46 -0800

David,

I think the key here is to go back to the definition of iSCSI data-type PDU 
(section 1.1), which states that an iSCSI data-type PDU is not delivered 
as-is to the remote iSCSI Layer since the associated data transfer is done 
without the participation of the remote iSCSI Layer.  So rewriting the text 
with the rationale of the definition in mind, here is a revised take:

RDMA Write, RDMA Read Request, and RDMA Read Response Messages are used for 
carrying control and all data information associated with the iSCSI 
data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs).  By definition (see 
section 1.1), an iSCSI data-type PDU is an iSCSI PDU that causes data 
transfer which is transparent to the remote iSCSI Layer.  As a result, the 
PDU itself is not delivered as-is to the remote iSCSI Layer.  In the case of 
R2Ts, the iSER layer at the target transforms the R2Ts into RDMA Read 
operations.  The solicited SCSI write data is sent by the iSER layer at the 
initiator using RDMA Read Response Messages instead of SCSI Data-out PDUs. 
This is not the case for unsolicited SCSI Write data, which is sent by the 
iSCSI layer as a SCSI Data-out PDU and is sent as-is by the iSER layer using 
a Send Message.  Because the iSCSI layer is invovled in the handling of the 
unsolicited SCSI write data, the SCSI Data-out PDU for unsolicited data is 
not considered an iSCSI data-type PDU.  See section 7.1 for more discussion 
of iSCSI data-type PDUs.

What do you think?

Mike

----- Original Message ----- 
From: [email-masked]
To: [email-masked] ; [email-masked]
Sent: Friday, December 09, 2011 2:52 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Mike,

For item 5, I think it's important that Section 2.3 item 4 make it clear 
that for iSER, the SCSI Data-out PDU is *not* a data-type PDU.  My proposed 
text was a bit weak on that point, but I didn't see it at all in your 
proposed text.

Here's another attempt at new text (combining text from both of us and 
editing further):

Unfortunately, this is still somewhat subtle and potentially confusing (the 
Data-out PDU is not a data-type PDU, rather the Data-out PDU is a 
control-type PDU ... huh??); the alternative is probably a serious rewrite 
to replace the terms "data-type PDU" and "control-type PDU" with terms that 
don't use "data" and "control", e.g., "RDMA-transfer PDU" and 
"message-transfer PDU".

What do you think?

Thanks,
--David

David,

Mike

----- Original Message ----- 

Hi Mike,

OLD

      on iSCSI data-type PDUs.

NEW

END

      means.

Thanks,
--David

David,

Mike

----- Original Message ----- 

To: [email-masked]

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: <[email-masked]>
Date: Sun, 11 Dec 2011 18:22:16 -0500

Mike,

I think this is getting to be too much text (thanks for your suggestions).

Upon further reflection, I'm ok with keeping the term "iSCSI data-type PDU", as that was used in RFC 5046.

How about the following shorter change?:

OLD
   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
      are used for carrying control and all data information associated
      with the iSCSI data-type PDUs.  See section 7.1 for more details
      on iSCSI data-type PDUs.
NEW
   4. RDMA Write, RDMA Read Request, and RDMA Read Response Messages
      are used for carrying control and all data information associated
      with the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs).
iSER does not use SCSI Data-Out PDUs for solicited data, and SCSI
Data-Out PDUs for unsolicited data are not treated as iSCSI data-type
PDUs by iSER because the data is unsolicited.  See section 7.1 for
more details on iSCSI data-type PDUs.
END

I think that summarizes the situation with SCSI Data-Out PDUs, and points to Section 7.1 for the details.

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Friday, December 09, 2011 7:25 PM
To: Black, David; [email-masked]
Subject: Re: [storm] WG Last Call - iSER - comments

David,

I think the key here is to go back to the definition of iSCSI data-type PDU (section 1.1), which states that an iSCSI data-type PDU is not delivered as-is to the remote iSCSI Layer since the associated data transfer is done without the participation of the remote iSCSI Layer.  So rewriting the text with the rationale of the definition in mind, here is a revised take:

RDMA Write, RDMA Read Request, and RDMA Read Response Messages are used for carrying control and all data information associated with the iSCSI data-type PDUs (i.e., SCSI Data-In PDUs and R2T PDUs).  By definition (see section 1.1), an iSCSI data-type PDU is an iSCSI PDU that causes data transfer which is transparent to the remote iSCSI Layer.  As a result, the PDU itself is not delivered as-is to the remote iSCSI Layer.  In the case of R2Ts, the iSER layer at the target transforms the R2Ts into RDMA Read operations.  The solicited SCSI write data is sent by the iSER layer at the initiator using RDMA Read Response Messages instead of SCSI Data-out PDUs.  This is not the case for unsolicited SCSI Write data, which is sent by the iSCSI layer as a SCSI Data-out PDU and is sent as-is by the iSER layer using a Send Message.  Because the iSCSI layer is invovled in the handling of the unsolicited SCSI write data, the SCSI Data-out PDU for unsolicited data is not considered an iSCSI data-type PDU.  See section 7.1 for more discussion of iSCSI data-type PDUs.

What do you think?

Mike

----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Friday, December 09, 2011 2:52 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Mike,

What do you think?

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, December 05, 2011 11:26 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Hi Mike,

OLD

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]>
Sent: Friday, December 02, 2011 4:00 PM
Subject: [storm] WG Last Call - iSER - comments

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: Michael Ko <[email-masked]>
Date: Mon, 12 Dec 2011 12:32:36 -0800

David,

"SCSI Data-Out PDUs for unsolicited data are not treated as iSCSI data-type 
PDUs by iSER because the data is unsolicited" seems like circular definition 
to me.  How about "SCSI Data-Out PDUs for unsolicited data are not treated 
as iSCSI data-type PDUs by iSER because RDMA is not used"?  This ties it 
back to the first sentence which states that RDMA is used with iSCSI 
data-type PDUs.

Mike
----- Original Message ----- 
From: [email-masked]
To: [email-masked] ; [email-masked]
Sent: Sunday, December 11, 2011 3:22 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Mike,

Upon further reflection, I'm ok with keeping the term "iSCSI data-type PDU", 
as that was used in RFC 5046.

OLD

      on iSCSI data-type PDUs.

NEW

END

I think that summarizes the situation with SCSI Data-Out PDUs, and points to 
Section 7.1 for the details.

Thanks,
--David

David,

What do you think?

Mike

----- Original Message ----- 

Mike,

What do you think?

Thanks,
--David

David,

Mike

----- Original Message ----- 

Hi Mike,

OLD

      on iSCSI data-type PDUs.

NEW

END

      means.

Thanks,
--David

David,

Mike

----- Original Message ----- 

To: [email-masked]

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: <[email-masked]>
Date: Mon, 12 Dec 2011 15:34:15 -0500

That works for me.

Thanks,
--David

From: Michael Ko [mailto:[email-masked]]
Sent: Monday, December 12, 2011 3:33 PM
To: Black, David; [email-masked]
Subject: Re: [storm] WG Last Call - iSER - comments

David,

"SCSI Data-Out PDUs for unsolicited data are not treated as iSCSI data-type PDUs by iSER because the data is unsolicited" seems like circular definition to me.  How about "SCSI Data-Out PDUs for unsolicited data are not treated as iSCSI data-type PDUs by iSER because RDMA is not used"?  This ties it back to the first sentence which states that RDMA is used with iSCSI data-type PDUs.

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Sunday, December 11, 2011 3:22 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Mike,

OLD

Thanks,
--David

David,

What do you think?

Mike

----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Friday, December 09, 2011 2:52 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Mike,

What do you think?

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, December 05, 2011 11:26 PM
Subject: RE: [storm] WG Last Call - iSER - comments

Hi Mike,

OLD

Thanks,
--David

David,

Mike
----- Original Message -----
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]>
Sent: Friday, December 02, 2011 4:00 PM
Subject: [storm] WG Last Call - iSER - comments

[mk] Done.

[mk] Done

[mk] Done

[mk] Done

[mk] Done.

Thanks,
--David

---

From: "Hemal Shah" <[email-masked]>
Date: Mon, 12 Dec 2011 17:06:07 -0800

David and Mike,

Below are my comments on the iSER draft.

Minor

1.      Section 1.1: iSCSI data-type PDU definition should clearly say at what layer the data transfer is happening. For example, the following sentence "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection." should be rephrased (added text in red) as "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection. I believe that the definition is already being revised based on David's comments.
2.      Section 1.1: The definition of iSCSI/iSER session needs to have further description. The current description is insufficient. Suggest to add the following to the definition "All connections of an iSCSI/iSER session are iSCSI/iSER connections."
3.      SNIC term is not defined but used in this draft. This needs to be addressed.
4.      Sections 2.4.3 and 2.4.4 use terms "sending side", "receiving side", "remote node", etc. It is confusing to use multiple terms (some of them are not defined in the draft) in these two sections. Suggest to use "Local Peer/Remote Peer" or "Data Source/Data Sink" terms consistently in these two sections.
5.      The use of term input qualifiers in Section 3 is misleading as we are actually referring to the input parameters for the operational primitives. Suggest to replace term "input qualifier" with "input parameter" in Section 3.
6.      Section 5.1.3: In the following sentence the second "iSER-ORD" should be replaced with "ORD". "Upon receiving the iSER Hello Message, the iSER Layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator."
7.      Section 9.2: Do WSV and RSV flags indicate the validity of WBO and RBO fields? The description of these flags should state it.
8.      Section 15 should be titled as "iWARP Message Formats for iSER".
9.      Sections 15.3-15.5 should be formatted as iWARP messages.
10.     I do not understand the reason behind removing the iWARP Message Format for RDMA Read Request, iWARP Message Format for Solicited SCSI Write Data, and iWARP Message Format for SCSI Response PDU in Section 15. They should be kept in Section 15.

Major
11.     According to the iSCSI spec, an iSCSI connection has an underlying TCP connection. Based on that, an iSER-assisted iSCSI connection requires an underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI connection has an underlying TCP connection. For iSER-assisted iSCSI connection over InfiniBand transports, there is no underlying TCP connection. In this case, what does an iSER-assisted iSCSI connection mean? What IB transport service types are used for the iSER-assisted iSCSI connection? The draft is not clear on what an iSER-assisted iSCSI connection mean for the InfiniBand transports. The draft needs to clearly define iSER-assisted iSCSI connection for InfiniBand transports and its mapping over underlying InfiniBand transport connections and transport service types.
12.     Section 4.1: "For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode." What is the meaning of this sentence? What message delivery capability is being referred here? Does this sentence infer that for IB transports, IB messages are allowed prior to enabling the iSER-assisted mode? The spec is not clear on how the iSCSI PDUs are transferred over IB connections prior to enabling iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted mode (I think it is the later)?
13.     Section 5.1 has the following text "If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) " What is the implication of re-establishing connection in TCP capable mode for InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, establishing a new TCP/IPoIB connection, and performing iSCSI login without RDMAExtensions key on that TCP connection? The spec does not describe this behavior clearly.
14.     Section 9.2: iSER header and iSER messages defined in Section 9.2 are not backward compatible with RFC 5046 defined iSER messages. I consider this as a significant issue. I suggest to add a single flag (or two flags) in iSCSI-control type PDU to indicate the presence of WBO and RBO by using one (or two) of the reserved bits. If this flag is set to 1, then the format is as defined in Figure 3 of this draft. If this flag is cleared, then the format of the iSER header for iSCSI control-type PDU is same as the one defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and HelloReply Messages to be identical to the formats defined in RFC5046. These proposed modifications to the header formats in Section 9.2 make this draft backward compatible with RFC5046 when WBO and RBO are not used.
15.     The headers in Sections 9.3 and 9.4 need to be reformatted if the change suggested in the previous bullet is made.
16.     We should add an appendix for "InfiniBand Message Formats for iSER" because they are different from "iWARP Message Formats for iSER".
17.     Solicited Event: RFC5046 describes the use of Send with Solicited Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) Messages. This draft removes the use of SendSE and SendSEInv Messages. What was the rationale behind this removal? The draft should allow the use of SendSE and SendSEInv unless we have good reasons not to.

Regards,

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, November 21, 2011 11:32 AM
To: [email-masked]
Subject: [storm] WG Last Call - iSER - through December 12
Importance: High

This message is to announce the STORM working group Last Call on the following document:

                  iSCSI Extensions for RDMA Specification
                       draft-ietf-storm-iser-05.txt

http://datatracker.ietf.org/doc/draft-ietf-storm-iser/

Last Call comments are due by Monday, December 12 at midnight Eastern time.

Please send all technical comments to the storm mailing list: [email-masked]
Editorial comments may be sent directly to the draft authors: [email-masked]

I'll even start this Working Group Last Call with an initial comment:

The iSER draft currently references RFC 3720 for iSCSI - that reference will need to be updated to at least the new consolidated iSCSI draft, and a reference to the iSCSI new features (SAM) draft should probably be added.

There's no need for a new version of the draft now, but a new version will be needed before the RFC publication request can be submitted.

Thanks in advance for the WG's prompt review. :-)

--David (storm WG co-chair)

---

From: <[email-masked]>
Date: Wed, 14 Dec 2011 09:43:46 -0500

Hemal,

Thank you for the detailed review.

The minor comments appear to all be editorial, so I'll leave them to Mike to resolve.

On the major comments:

Items 11 and 12 are the same issue, as I also found item 11, and the text quoted in item 12 was supposed to resolve it.  Obviously, that text is not sufficiently clear ;-), so please work with Mike to come up with a clearer paragraph.

Item 13 looks like a useful clarification - the non-TCP connection may have to be torn down in order to establish a TCP connection, and IB is an example where there is necessary.

Items 14 and 15 represent a deliberate change because the implementations did something different from what's in RFC 5046, as noted in Appendix A (Section 14).

   8. Added two 64-bit fields in iSER header in section 9.2 for the Read
     Base Offset and the Write Base Offset to accommodate a non-zero
     Base Offset.  This allows one implementation such as the OFED
     stack to be used in both the Infiniband and the iWARP environment.
     Changes were made in the definition of Base Offset, Advertisement,
     and Tagged Buffer.  Changes were also made in sections 2.4.1, 2.5,
     2.6, 7.3.1, 7.3.3, 7.3.5, 7.3.6, 9.1, 9.3, 9.4, 9.5.1, and 9.5.2.
     This change is not backward compatible with RFC 5046.

I would suggest changing the last sentence to:

     This change is not backward compatible with RFC 5046, but is part
     of all known implementations of iSER at the time this document was
     developed.

If you know of an implementation that did *NOT* do this, please speak up now!

Item 16 - I would prefer not to add the iSER InfiniBand message formats to this draft, but instead leave them to the IBTA to document.  OTOH, if there's no good documentation of these for InfiniBand, they could be added.  What's the state of IB documentation for iSER?

Item 17 - It looks like this is covered by item 9 in Annex A:

   9. Remove iWARP specific behavior.  Changes were made in the
     definition section on RDMA Operation and Send Message Type.
     Changes affecting Send with Invalidate were made in sections
     2.4.1, 2.5, 2.6, 4.1, and 7.3.2.  Changes affecting Terminate were
     made in sections 10.1.2.1 and 10.1.2.2.  Changes were made in
     section 15 to remove iWARP headers.

The notion of Solicited Event (SE) is iWARP-specific; the generalization above seems useful.  OTOH, RFC 5046 contained requirements for use of SendSE and SendInvSE, and those requirements appear to have been removed.  I believe that they need to be restored, possibly in a new section that is specifically about iSER over iWARP.

Thanks,
--David

From: Hemal Shah [mailto:[email-masked]]
Sent: Monday, December 12, 2011 8:06 PM
To: Black, David; [email-masked]
Subject: RE: WG Last Call - iSER - through December 12 - comments

David and Mike,

Minor

Major

 1.  According to the iSCSI spec, an iSCSI connection has an underlying TCP connection. Based on that, an iSER-assisted iSCSI connection requires an underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI connection has an underlying TCP connection. For iSER-assisted iSCSI connection over InfiniBand transports, there is no underlying TCP connection. In this case, what does an iSER-assisted iSCSI connection mean? What IB transport service types are used for the iSER-assisted iSCSI connection? The draft is not clear on what an iSER-assisted iSCSI connection mean for the InfiniBand transports. The draft needs to clearly define iSER-assisted iSCSI connection for InfiniBand transports and its mapping over underlying InfiniBand transport connections and transport service types.
 2.  Section 4.1: "For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode." What is the meaning of this sentence? What message delivery capability is being referred here? Does this sentence infer that for IB transports, IB messages are allowed prior to enabling the iSER-assisted mode? The spec is not clear on how the iSCSI PDUs are transferred over IB connections prior to enabling iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted mode (I think it is the later)?
 3.  Section 5.1 has the following text "If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) " What is the implication of re-establishing connection in TCP capable mode for InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, establishing a new TCP/IPoIB connection, and performing iSCSI login without RDMAExtensions key on that TCP connection? The spec does not describe this behavior clearly.
 4.  Section 9.2: iSER header and iSER messages defined in Section 9.2 are not backward compatible with RFC 5046 defined iSER messages. I consider this as a significant issue. I suggest to add a single flag (or two flags) in iSCSI-control type PDU to indicate the presence of WBO and RBO by using one (or two) of the reserved bits. If this flag is set to 1, then the format is as defined in Figure 3 of this draft. If this flag is cleared, then the format of the iSER header for iSCSI control-type PDU is same as the one defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and HelloReply Messages to be identical to the formats defined in RFC5046. These proposed modifications to the header formats in Section 9.2 make this draft backward compatible with RFC5046 when WBO and RBO are not used.
 5.  The headers in Sections 9.3 and 9.4 need to be reformatted if the change suggested in the previous bullet is made.
 6.  We should add an appendix for "InfiniBand Message Formats for iSER" because they are different from "iWARP Message Formats for iSER".
 7.  Solicited Event: RFC5046 describes the use of Send with Solicited Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) Messages. This draft removes the use of SendSE and SendSEInv Messages. What was the rationale behind this removal? The draft should allow the use of SendSE and SendSEInv unless we have good reasons not to.

Regards,

Hemal

---

From: Michael Ko <[email-masked]>
Date: Wed, 14 Dec 2011 13:37:01 -0800

Hemal,

Thank you for the extensive reivew.  My comments are inline.

Mike
----- Original Message ----- 
From: Hemal Shah
To: [email-masked]' ; [email-masked]
Sent: Monday, December 12, 2011 5:06 PM
Subject: Re: [storm] WG Last Call - iSER - through December 12 - comments

David and Mike,

Minor

  a.. Section 1.1: iSCSI data-type PDU definition should clearly say at what 
layer the data transfer is happening. For example, the following sentence 
"An iSCSI data-type PDU is defined as an iSCSI PDU that causes data 
transfer, transparent to the remote iSCSI Layer, to take place between the 
peer iSCSI nodes on a full feature phase iSCSI connection." should be 
rephrased (added text in red) as "An iSCSI data-type PDU is defined as an 
iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, 
transparent to the remote iSCSI Layer, to take place between the peer iSCSI 
nodes on a full feature phase iSCSI connection. I believe that the 
definition is already being revised based on David's comments.  [mk] OK if 
no one else objects.
  b.. Section 1.1: The definition of iSCSI/iSER session needs to have 
further description. The current description is insufficient. Suggest to add 
the following to the definition "All connections of an iSCSI/iSER session 
are iSCSI/iSER connections."  [mk] OK if no one else objects.
  c.. SNIC term is not defined but used in this draft. This needs to be 
addressed.  [mk] Simpler just to delete the reference to SNIC.
  d.. Sections 2.4.3 and 2.4.4 use terms "sending side", "receiving side", 
"remote node", etc. It is confusing to use multiple terms (some of them are 
not defined in the draft) in these two sections. Suggest to use "Local 
Peer/Remote Peer" or "Data Source/Data Sink" terms consistently in these two 
sections.  [mk] OK if no one else objects.
  e.. The use of term input qualifiers in Section 3 is misleading as we are 
actually referring to the input parameters for the operational primitives. 
Suggest to replace term "input qualifier" with "input parameter" in Section 
3.  [mk] "Qualifier" is a term used in the Datamover Architecture document 
referenced in the first paragraph of section 3.  iSER uses the abstract 
model as defined in the Datamover Architecture.
  f.. Section 5.1.3: In the following sentence the second "iSER-ORD" should 
be replaced with "ORD". "Upon receiving the iSER Hello Message, the iSER 
Layer at the target MUST set the iSER-ORD value to the minimum of the 
iSER-ORD value at the target and the iSER-IRD value declared by the 
initiator."  [mk] iSER-ORD is a variable while ORD is a concept.  Its usage 
here is correct.  See the definition in section 1.1 on iSER-ORD and ORD.
  g.. Section 9.2: Do WSV and RSV flags indicate the validity of WBO and RBO 
fields? The description of these flags should state it.  [mk] OK.
  h.. Section 15 should be titled as "iWARP Message Formats for iSER".  [mk] 
Current title is correct.
  i.. Sections 15.3-15.5 should be formatted as iWARP messages.  [mk] The 
intent is to show the iSER message format for the generic RCaP, not just 
iWARP.
  j.. I do not understand the reason behind removing the iWARP Message 
Format for RDMA Read Request, iWARP Message Format for Solicited SCSI Write 
Data, and iWARP Message Format for SCSI Response PDU in Section 15. They 
should be kept in Section 15.  [mk] The intent is to show the iSER message 
format for the generic RCaP, not just iWARP.

Major
  k.. According to the iSCSI spec, an iSCSI connection has an underlying TCP 
connection. Based on that, an iSER-assisted iSCSI connection requires an 
underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI 
connection has an underlying TCP connection. For iSER-assisted iSCSI 
connection over InfiniBand transports, there is no underlying TCP 
connection. In this case, what does an iSER-assisted iSCSI connection mean? 
What IB transport service types are used for the iSER-assisted iSCSI 
connection? The draft is not clear on what an iSER-assisted iSCSI connection 
mean for the InfiniBand transports. The draft needs to clearly define 
iSER-assisted iSCSI connection for InfiniBand transports and its mapping 
over underlying InfiniBand transport connections and transport service 
types.  [mk] The support for iSER in Infiniband can be found in Annex A12 of 
the Infiniband Architecture Specification which appeared as an Informative 
Reference in section 13.2.
  l.. Section 4.1: "For a transport layer that provides message delivery 
capability such as [IB], the RCaP implementation supports the use of the 
messaging capability by the iSCSI Layer directly for the Login phase after 
connection establishment before enabling iSER-assisted mode." What is the 
meaning of this sentence? What message delivery capability is being referred 
here? Does this sentence infer that for IB transports, IB messages are 
allowed prior to enabling the iSER-assisted mode? The spec is not clear on 
how the iSCSI PDUs are transferred over IB connections prior to enabling 
iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB 
connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted 
mode (I think it is the later)?  [mk] The support for iSER in Infiniband can 
be found in Annex A12 of the Infiniband Architecture Specification which 
appeared as an Informative Reference in section 13.2.
  m.. Section 5.1 has the following text "If the RDMAExtensions key is not 
negotiated to Yes, then for some RCaP implementation (such as [IB]), the 
connection may need to be re-established in TCP capable mode.  (For 
InfiniBand this will require an [IPoIB] type connection.) " What is the 
implication of re-establishing connection in TCP capable mode for 
InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, 
establishing a new TCP/IPoIB connection, and performing iSCSI login without 
RDMAExtensions key on that TCP connection? The spec does not describe this 
behavior clearly.  The support for iSER in Infiniband can be found in Annex 
A12 of the Infiniband Architecture Specification which appeared as an 
Informative Reference in section 13.2.
  n.. Section 9.2: iSER header and iSER messages defined in Section 9.2 are 
not backward compatible with RFC 5046 defined iSER messages. I consider this 
as a significant issue. I suggest to add a single flag (or two flags) in 
iSCSI-control type PDU to indicate the presence of WBO and RBO by using one 
(or two) of the reserved bits. If this flag is set to 1, then the format is 
as defined in Figure 3 of this draft. If this flag is cleared, then the 
format of the iSER header for iSCSI control-type PDU is same as the one 
defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and 
HelloReply Messages to be identical to the formats defined in RFC5046. These 
proposed modifications to the header formats in Section 9.2 make this draft 
backward compatible with RFC5046 when WBO and RBO are not used.  [mk] Will 
change the text in section 14 as suggested by David Black.
  o.. The headers in Sections 9.3 and 9.4 need to be reformatted if the 
change suggested in the previous bullet is made.
  p.. We should add an appendix for "InfiniBand Message Formats for iSER" 
because they are different from "iWARP Message Formats for iSER".  [mk] The 
support for iSER in Infiniband can be found in Annex A12 of the Infiniband 
Architecture Specification which appeared as an Informative Reference in 
section 13.2.
  q.. Solicited Event: RFC5046 describes the use of Send with Solicited 
Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) 
Messages. This draft removes the use of SendSE and SendSEInv Messages. What 
was the rationale behind this removal? The draft should allow the use of 
SendSE and SendSEInv unless we have good reasons not to.  [mk] The intent is 
to describe how iSER is supposed to work over a generic RCaP.  For iWARP, I 
can restore the references to SendSE and SendSEInv where applicable by 
including them inside parenthesis.

Regards,

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
[email-masked]
Sent: Monday, November 21, 2011 11:32 AM
To: [email-masked]
Subject: [storm] WG Last Call - iSER - through December 12
Importance: High

This message is to announce the STORM working group Last Call on the 
following document:

Please send all technical comments to the storm mailing list: [email-masked]
Editorial comments may be sent directly to the draft authors: 
[email-masked]

The iSER draft currently references RFC 3720 for iSCSI - that reference will 
need to be updated to at least the new consolidated iSCSI draft, and a 
reference to the iSCSI new features (SAM) draft should probably be added.

There's no need for a new version of the draft now, but a new version will 
be needed before the RFC publication request can be submitted.

--------------------------------------------------------------------------------

---

From: "Hemal Shah" <[email-masked]>
Date: Wed, 21 Dec 2011 11:00:10 -0800

Mike,

Thanks! I replied to David's reply to my original message. I'm fine with your responses to 1-10 below. For 11-17, please look at my response to David's reply. Let us resolve those items using that email thread.

Hemal

From: Michael Ko [mailto:[email-masked]]
Sent: Wednesday, December 14, 2011 1:37 PM
To: Hemal Shah; [email-masked]; [email-masked]
Subject: Re: [storm] WG Last Call - iSER - through December 12 - comments

Hemal,

Mike
----- Original Message -----
From: Hemal Shah<mailto:[email-masked]>
To: [email-masked]'<mailto:[email-masked]'> ; [email-masked]<mailto:[email-masked]>
Sent: Monday, December 12, 2011 5:06 PM
Subject: Re: [storm] WG Last Call - iSER - through December 12 - comments

David and Mike,

Minor

 1.  Section 1.1: iSCSI data-type PDU definition should clearly say at what layer the data transfer is happening. For example, the following sentence "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection." should be rephrased (added text in red) as "An iSCSI data-type PDU is defined as an iSCSI PDU that causes data transfer via RDMA operations at the iSER layer, transparent to the remote iSCSI Layer, to take place between the peer iSCSI nodes on a full feature phase iSCSI connection. I believe that the definition is already being revised based on David's comments.  [mk] OK if no one else objects.
 2.  Section 1.1: The definition of iSCSI/iSER session needs to have further description. The current description is insufficient. Suggest to add the following to the definition "All connections of an iSCSI/iSER session are iSCSI/iSER connections."  [mk] OK if no one else objects.
 3.  SNIC term is not defined but used in this draft. This needs to be addressed.  [mk] Simpler just to delete the reference to SNIC.
 4.  Sections 2.4.3 and 2.4.4 use terms "sending side", "receiving side", "remote node", etc. It is confusing to use multiple terms (some of them are not defined in the draft) in these two sections. Suggest to use "Local Peer/Remote Peer" or "Data Source/Data Sink" terms consistently in these two sections.  [mk] OK if no one else objects.
 5.  The use of term input qualifiers in Section 3 is misleading as we are actually referring to the input parameters for the operational primitives. Suggest to replace term "input qualifier" with "input parameter" in Section 3.  [mk] "Qualifier" is a term used in the Datamover Architecture document referenced in the first paragraph of section 3.  iSER uses the abstract model as defined in the Datamover Architecture.
 6.  Section 5.1.3: In the following sentence the second "iSER-ORD" should be replaced with "ORD". "Upon receiving the iSER Hello Message, the iSER Layer at the target MUST set the iSER-ORD value to the minimum of the iSER-ORD value at the target and the iSER-IRD value declared by the initiator."  [mk] iSER-ORD is a variable while ORD is a concept.  Its usage here is correct.  See the definition in section 1.1 on iSER-ORD and ORD.
 7.  Section 9.2: Do WSV and RSV flags indicate the validity of WBO and RBO fields? The description of these flags should state it.  [mk] OK.
 8.  Section 15 should be titled as "iWARP Message Formats for iSER".  [mk] Current title is correct.
 9.  Sections 15.3-15.5 should be formatted as iWARP messages.  [mk] The intent is to show the iSER message format for the generic RCaP, not just iWARP.
 10. I do not understand the reason behind removing the iWARP Message Format for RDMA Read Request, iWARP Message Format for Solicited SCSI Write Data, and iWARP Message Format for SCSI Response PDU in Section 15. They should be kept in Section 15.  [mk] The intent is to show the iSER message format for the generic RCaP, not just iWARP.

Major

 1.  According to the iSCSI spec, an iSCSI connection has an underlying TCP connection. Based on that, an iSER-assisted iSCSI connection requires an underlying TCP connection to be present. For iWARP, an iSER-assisted iSCSI connection has an underlying TCP connection. For iSER-assisted iSCSI connection over InfiniBand transports, there is no underlying TCP connection. In this case, what does an iSER-assisted iSCSI connection mean? What IB transport service types are used for the iSER-assisted iSCSI connection? The draft is not clear on what an iSER-assisted iSCSI connection mean for the InfiniBand transports. The draft needs to clearly define iSER-assisted iSCSI connection for InfiniBand transports and its mapping over underlying InfiniBand transport connections and transport service types.  [mk] The support for iSER in Infiniband can be found in Annex A12 of the Infiniband Architecture Specification which appeared as an Informative Reference in section 13.2.
 2.  Section 4.1: "For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode." What is the meaning of this sentence? What message delivery capability is being referred here? Does this sentence infer that for IB transports, IB messages are allowed prior to enabling the iSER-assisted mode? The spec is not clear on how the iSCSI PDUs are transferred over IB connections prior to enabling iSER-assisted mode. Are iSCSI PDUs transferred over iSCSI/TCP/IPoIB connection or iSCSI/IB Reliable Connection prior to enabling iSER-assisted mode (I think it is the later)?  [mk] The support for iSER in Infiniband can be found in Annex A12 of the Infiniband Architecture Specification which appeared as an Informative Reference in section 13.2.
 3.  Section 5.1 has the following text "If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) " What is the implication of re-establishing connection in TCP capable mode for InfiniBand? Does it require tearing down the exisiting iSCSI/IB connection, establishing a new TCP/IPoIB connection, and performing iSCSI login without RDMAExtensions key on that TCP connection? The spec does not describe this behavior clearly.  The support for iSER in Infiniband can be found in Annex A12 of the Infiniband Architecture Specification which appeared as an Informative Reference in section 13.2.
 4.  Section 9.2: iSER header and iSER messages defined in Section 9.2 are not backward compatible with RFC 5046 defined iSER messages. I consider this as a significant issue. I suggest to add a single flag (or two flags) in iSCSI-control type PDU to indicate the presence of WBO and RBO by using one (or two) of the reserved bits. If this flag is set to 1, then the format is as defined in Figure 3 of this draft. If this flag is cleared, then the format of the iSER header for iSCSI control-type PDU is same as the one defined in RFC5046. I strongly suggest to keep the formats of iSER Hello and HelloReply Messages to be identical to the formats defined in RFC5046. These proposed modifications to the header formats in Section 9.2 make this draft backward compatible with RFC5046 when WBO and RBO are not used.  [mk] Will change the text in section 14 as suggested by David Black.
 5.  The headers in Sections 9.3 and 9.4 need to be reformatted if the change suggested in the previous bullet is made.
 6.  We should add an appendix for "InfiniBand Message Formats for iSER" because they are different from "iWARP Message Formats for iSER".  [mk] The support for iSER in Infiniband can be found in Annex A12 of the Infiniband Architecture Specification which appeared as an Informative Reference in section 13.2.
 7.  Solicited Event: RFC5046 describes the use of Send with Solicited Event (SendSE) and Send with Solicited Event and Invalidate (SendSEInv) Messages. This draft removes the use of SendSE and SendSEInv Messages. What was the rationale behind this removal? The draft should allow the use of SendSE and SendSEInv unless we have good reasons not to.  [mk] The intent is to describe how iSER is supposed to work over a generic RCaP.  For iWARP, I can restore the references to SendSE and SendSEInv where applicable by including them inside parenthesis.

Regards,

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Monday, November 21, 2011 11:32 AM
To: [email-masked]
Subject: [storm] WG Last Call - iSER - through December 12
Importance: High

________________________________
_______________________________________________
storm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/storm

---

From: "Hemal Shah" <[email-masked]>
Date: Wed, 21 Dec 2011 10:57:27 -0800

David,

See my reply below!

Hemal

From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, December 14, 2011 6:44 AM
To: Hemal Shah; [email-masked]
Subject: RE: WG Last Call - iSER - through December 12 - comments

Hemal,

On the major comments:

Items 11 and 12 are the same issue, as I also found item 11, and the text quoted in item 12 was supposed to resolve it.  Obviously, that text is not sufficiently clear ;-), so please work with Mike to come up with a clearer paragraph.
[Hemal]
For Item 11, I suggest the addition of the following text to the definition of iSER-assisted iSCSI connection:

An iSCSI connection that is not iSER-assisted always maps on to a TCP connection at the transport level. But, an iSER-assisted iSCSI connection may not have an underlying TCP connection. For an iWARP based RCaP implementation, an iSER-assisted iSCSI connection has an underlying TCP connection. But, for an iSER-assisted iSCSI connection over an InfiniBand [IB] based RCaP implementation, there is no underlying TCP connection. For an InfiniBand [IB] based RCaP implementation, an iSER-assisted iSCSI connection is directly mapped onto InfiniBand RC channel.

[Hemal] For Item 12, I suggest the following changes to Section 4.1.

OLD
"For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode."

NEW
"For a transport layer that provides message delivery capability such as [IB], the RCaP implementation supports the use of the messaging capability by the iSCSI Layer directly for the Login phase after connection establishment before enabling iSER-assisted mode. For an InfiniBand [IB] based RCaP implementation, the iSCSI Layer uses IB messages to transfer iSCSI PDUs for the Login phase after connection establishment before enabling iSER-assisted mode."

Item 13 looks like a useful clarification - the non-TCP connection may have to be torn down in order to establish a TCP connection, and IB is an example where there is necessary.
[Hemal] I suggest the following modification to the text in Section 5.1.

OLD
"If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the connection may need to be re-established in TCP capable mode.  (For InfiniBand this will require an [IPoIB] type connection.) "

NEW
"If the RDMAExtensions key is not negotiated to Yes, then for some RCaP implementation (such as [IB]), the existing connection may need to be torn down and a new connection may need to be established in TCP capable mode.  (For InfiniBand, this will require an [IPoIB] type connection.) "

If you know of an implementation that did *NOT* do this, please speak up now!
[Hemal] I like your suggested text above.

Item 16 - I would prefer not to add the iSER InfiniBand message formats to this draft, but instead leave them to the IBTA to document.  OTOH, if there's no good documentation of these for InfiniBand, they could be added.  What's the state of IB documentation for iSER?
[Hemal] I'm fine with it. I suggest that we add a reference to look at InfiniBand documents for iSER InfiniBand message formats. Mike should be able to answer your question.

The notion of Solicited Event (SE) is iWARP-specific; the generalization above seems useful.  OTOH, RFC 5046 contained requirements for use of SendSE and SendInvSE, and those requirements appear to have been removed.  I believe that they need to be restored, possibly in a new section that is specifically about iSER over iWARP.
[Hemal] I agree that these requirements need to be restored and added as a new section.

Thanks,
--David

David and Mike,

Minor

Major

Regards,

Hemal

---

From: <[email-masked]>
Date: Wed, 21 Dec 2011 17:17:48 -0500

Hemal and Mike,

This looks like progress.  The text changes mostly look ok, but I have a couple of comments:

(1) Please always introduce InfiniBand as an example - prefixing "e.g.," or "for example" usually suffices.  I'd like to avoid having an IB spec as a normative reference.  This also affects some of Mike's existing text.  As part of this, I'd rewrite the last two sentences of Hemal's new text for item 11 as:

But, for an iSER-assisted iSCSI connection over some RCaP implementations (e.g., InfiniBand), there is no underlying TCP connection. For the specific example of an InfiniBand [IB] based RCaP implementation, an iSER-assisted iSCSI connection is directly mapped onto InfiniBand RC channel.

(2) It looks like the new iWARP-specific text to restore the requirements for SendSE and SendInvSE requirements is the major work item in this.

Thanks,
--David

From: Hemal Shah [mailto:[email-masked]]
Sent: Wednesday, December 21, 2011 1:57 PM
To: Black, David; [email-masked]
Subject: RE: WG Last Call - iSER - through December 12 - comments

David,

See my reply below!

Hemal

Hemal,

On the major comments:

OLD

NEW

OLD

NEW

Thanks,
--David

David and Mike,

Minor

Major

Regards,

Hemal

---

From: "Hemal Shah" <[email-masked]>
Date: Wed, 21 Dec 2011 14:23:05 -0800

Thanks David! I agree with your points #1 and #2 below.

Hemal

From: [email-masked] [mailto:[email-masked]]
Sent: Wednesday, December 21, 2011 2:18 PM
To: Hemal Shah; [email-masked]
Subject: RE: WG Last Call - iSER - through December 12 - comments

Hemal and Mike,

Thanks,
--David

David,

See my reply below!

Hemal

Hemal,

On the major comments:

OLD

NEW

OLD

NEW

Thanks,
--David

David and Mike,

Minor

Major

Regards,

Hemal