Subject: [rddp] verb spec clarification

From: Patricio Kaplan <[email-masked]>
Date: Mon, 28 Mar 2005 11:06:31 -0800

Section 9.2.5.1 states that QPs have Inbound RDMA Read enable and
Inbound RDMA Write and inbound RDMA Read Response enable.
 
Does the  inbound RDMA Read Response enable check apply to a
zero length read response?

Thanks,
-Patricio

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Mon, 28 Mar 2005 12:33:09 -0700

Do you mean the RDMAC verbs document - I can't even seem to find it on
the IETF ID site right now.

If so - asking on the RDMAC mailing list might work better (all though
many of the people should be in both places)

Bill

> -------- Original Message --------
> Subject: [rddp] verb spec clarification
> From: Patricio Kaplan <[email-masked]>
> Date: Mon, March 28, 2005 11:06 am
> To: [email-masked]
>

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 28 Mar 2005 13:58:14 -0800

The verbs are not an IETF document. Under the current consensus
they will not be.

However, the security draft does assume that inbound RDMA
Read Requests do consume a resource, and that the ULP should
be tracking these resources.

In that spirit, the length of the payload for an RDMA Read
does not affect its ordering within a sequence of RDMA Reads.
Therefore it requires the same resource support as any other
RDMA Read request, and would count toward the same credit limit.
The fact that there is not payload, and no validation of the
data source STag, does not mean it is not a "real" RDMA Read.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Patricio Kaplan
> Sent: Monday, March 28, 2005 11:07 AM
> To: [email-masked]
> Section 9.2.5.1 states that QPs have Inbound RDMA Read enable 
> and Inbound RDMA Write and inbound RDMA Read Response enable.
>

---

From: [email-masked]
Date: Tue, 29 Mar 2005 17:23:55 -0700

Caitlin,

Yes, the RDMA Read Response for a zero length read will obey the ordering rules. That is what allows it to be used as a fence which is its primary purpose. The only checks turned off (in the sections Paul quoted) are those that check the appropriateness of the memory region/window targeted since it doesn't really access the region/window.

A zero-length RDMA Read needs to respect the negotiated RDMA Read queue depth. If one arrives and there is no buffer it will cause an error. The resource checks related to that behavior apply.

Regards,
Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Caitlin Bestler
Sent: Tuesday, 29 March, 2005 6:39 AM
To: Culley, Paul
Cc: Patricio Kaplan; [email-masked]
Subject: Re: [rddp] verb spec clarification

Culley, Paul wrote:
> For Verbs, The answer is no.
> 
> Section 7.6.3 pg 112 near top
> "If the length of the access is zero, the RI MUST NOT perform any of the
> above checks on the Memory Region."
> 
> Section 7.10.6.2 pg 124, 2/3 down
> "If the length of the access is zero, the RI MUST NOT perform any of the
> above checks on the Memory Window."
> 
> Paul R. Culley
> HP Fellow
> 281-514-5543
>  

Putting aside any overly broad use of the phrase "above checks",
would you agree that the *only* thing special about zero length
reads is that the STag is not validated?

Specifically, the RDMA Read Response will still be ordered just
as the response to any other RDMA Read Request, correct?

Therefore, it will consume a resource (on either a connection
or wider scale). That resource SHOULD/MUST be properly accounted
for by *some* mechanism.

I can think of nothing in the wire specifications that would
justify suppressing of normal resource checking for zero length
reads. The wire protocol does not specify *how* the local
interface tracks resources, but the security draft requires
that they be managed somehow whenever they are used across
multiple connections or are in any way a resource more
limited than just host memory.

---

From: [email-masked]
Date: Tue, 29 Mar 2005 18:07:31 -0700

I think Paul missed the context of the question. Not surprising because the question is a little confusing:
"Section 9.2.5.1 states that QPs have Inbound RDMA Read enable and
Inbound RDMA Write and inbound RDMA Read Response enable.
 
Does the inbound RDMA Read Response enable check apply to a
zero length read response?"

The sections Paul mentioned don't address this question.

There is no RDMA Read Response enable check. Verbs seems to lack any description of the effect of these enables.

RDMA Read Response enable is a feature of queue creation. Presumably if RDMA Read Response handling is not enabled, there will be no resources provided for initiating RDMA Reads, i.e. ORD will be zero. Attempting to initiate an RDMA Read will fail.

These enables are not described very much and they may seem redundant with setting IRD and ORD to zero. A modify QP can change IRD and ORD. Setting up the QP with Inbound RDMA Read Response enabled and ORD zero would allow a later attempt to modify the QP to increase ORD. My expectation is that when a QP was set up with Inbound RDMA Read Response disabled, the RNIC can assume that RDMA Read initiation will never be allowed and any attempt to allocate ORD depth could be failed. This is the way I think it should work, but the reaction of an RNIC to a Create QP with Inbound RDMA write and RDMA Read Response disabled and ORD non-zero or a Modify QP for non-zero ORD when this is disabled are not described.

It also isn't clear why Inbound RDMA Write and inbound RDMA Read response share a single enable. They require different resources since the RDMA Read Response requires ORD depth. They aren't always needed together. An iSCSI target will receive inbound RDMA Writes and will not receive RDMA Read responses. An iSCSI initiator has the reverse situation. 

Regards,
Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Caitlin Bestler
Sent: Tuesday, 29 March, 2005 7:41 AM
To: Culley, Paul
Cc: Patricio Kaplan; [email-masked]
Subject: Re: [rddp] verb spec clarification

> above checks on the Memory Window."

Additional point: checking IRD credits is a check on the connection(QP)
not on the Memory Region/Memory Window.

---

From: "Culley, Paul" <[email-masked]>
Date: Wed, 30 Mar 2005 07:50:34 -0600

I've always thought that "Inbound RDMA Write and inbound RDMA Read
Response" were the same thing (one bit) for purposes of the QP enable.
Basically an inbound RDMA Read Response IS an inbound RDMA Write (same
DDP opcode).  The idea for this enable was inherited from VI.

The sections I referenced DO address this; on page 111 for Memory Region
checks the following is stated:

"When a local or remote operation attempts to access a registered Memory
Region, the RI MUST ensure that: 
...
The Access Rights of the QP allow the type of access being performed by
the operation,"

And the zero length check (stated in thread below) over-rides this.

There is similar text under the Memory Window section.

Paul R. Culley
HP Fellow
281-514-5543

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Tuesday, March 29, 2005 7:08 PM
> To: [email-masked]; Culley, Paul
> Cc: [email-masked]; [email-masked]
> Subject: RE: [rddp] verb spec clarification
> 
> I think Paul missed the context of the question. Not 
> surprising because the question is a little confusing:
> "Section 9.2.5.1 states that QPs have Inbound RDMA Read 
> enable and Inbound RDMA Write and inbound RDMA Read Response enable.
>  
> There is no RDMA Read Response enable check. Verbs seems to 
> lack any description of the effect of these enables.
> 
> RDMA Read Response enable is a feature of queue creation. 
> Presumably if RDMA Read Response handling is not enabled, 
> there will be no resources provided for initiating RDMA 
> Reads, i.e. ORD will be zero. Attempting to initiate an RDMA 
> Read will fail.
> 
> These enables are not described very much and they may seem 
> redundant with setting IRD and ORD to zero. A modify QP can 
> change IRD and ORD. Setting up the QP with Inbound RDMA Read 
> Response enabled and ORD zero would allow a later attempt to 
> modify the QP to increase ORD. My expectation is that when a 
> QP was set up with Inbound RDMA Read Response disabled, the 
> RNIC can assume that RDMA Read initiation will never be 
> allowed and any attempt to allocate ORD depth could be 
> failed. This is the way I think it should work, but the 
> reaction of an RNIC to a Create QP with Inbound RDMA write 
> and RDMA Read Response disabled and ORD non-zero or a Modify 
> QP for non-zero ORD when this is disabled are not described.
> 
> It also isn't clear why Inbound RDMA Write and inbound RDMA 
> Read response share a single enable. They require different 
> resources since the RDMA Read Response requires ORD depth. 
> They aren't always needed together. An iSCSI target will 
> receive inbound RDMA Writes and will not receive RDMA Read 
> responses. An iSCSI initiator has the reverse situation. 
> 
> Regards,
> Pat
> 
> From: [email-masked] [mailto:[email-masked]]On 
> Behalf Of Caitlin Bestler
> > "If the length of the access is zero, the RI MUST NOT 
> perform any of 
> > the above checks on the Memory Region."
> > 
> > "If the length of the access is zero, the RI MUST NOT 
> perform any of 
> > the above checks on the Memory Window."
> > 
> 
> Additional point: checking IRD credits is a check on the 
> connection(QP) not on the Memory Region/Memory Window.
>

---

From: "Caitlin Bestler" <[email-masked]>
Date: Wed, 30 Mar 2005 10:10:02 -0800

First and foremost, I want to be certain that everyone agrees
that zero length RDMA Reads must be accounted for in IRD/ORD
resource tracking. They are not processed out-of-order and
cannot be considered as some sort of "out-of-band" message.

For most local interfaces the IRD and ORD limits are set
locally when the QP is created and MAY be modified. As far
as IETF specifications are concerned, the existence of some
sort of IRD/ORD tracking is assumed. However, there is no
mandate that it be part of the local interface. It is merely
difficult to imagine complying with some of the security
requirements without doing so.

That leaves the question as to what was meant by having
both the enable and the IRD count in the earlier proposed
verbs draft. While techincally off topic, I think this is
of general interest, so I'd like to comment.

I have seen many translations of those verb requirements
to APIs that treated the enable as redundant, and merely
enabled/disabled "inbound RDMA Read capability" based on
whether the IRD was zero or non-zero.

Pat's earlier comment suggested that the enable bit was
a hint that the IRD would be modified to a non-zero value
at a later time and therefore the RI should ensure that
the appropriate data structures were setup, even if credit
was not granted. I can see that argument, but I would recommend
applications instead create the QP with the maximum IRD that 
they intend to use, and then use modify (before connecting)
to lower the limit to block excess incoming RDMA Reads.

Paul's response seems to imply that you could create a non-zero
IRD and *not* set the enable flag and that the result would be
that *only* zero length RDMA Reads would be allowed.

If that was the intent it is sufficiently unclear that I would
strongly recommend that application developers not rely on that
method. If you only want to enable zero-length RDMA Reads on a
given connection, simply do not enable any STags on that connection.

That capability clearly exists, and the ability to ensure that
no STags are valid on a connection is explicit in IETF specs.

> Behalf Of Culley, Paul
> Sent: Wednesday, March 30, 2005 5:51 AM
> To: [email-masked]; [email-masked]
> I've always thought that "Inbound RDMA Write and inbound RDMA 
> Read Response" were the same thing (one bit) for purposes of 
> the QP enable.
> Basically an inbound RDMA Read Response IS an inbound RDMA 
> Write (same DDP opcode).  The idea for this enable was 
> inherited from VI.
> 
> The sections I referenced DO address this; on page 111 for 
> Memory Region checks the following is stated:
> 
> "When a local or remote operation attempts to access a 
> registered Memory Region, the RI MUST ensure that: 
> ...
> The Access Rights of the QP allow the type of access being 
> performed by the operation,"
> 
> > I think Paul missed the context of the question. Not surprising 
> > because the question is a little confusing:
> > Does the inbound RDMA Read Response enable check apply to a zero 
> > length read response?"
> > 
> lack any 
> > description of the effect of these enables.
> > 
> there will 
> > be no resources provided for initiating RDMA Reads, i.e. 
> ORD will be 
> > zero. Attempting to initiate an RDMA Read will fail.
> > 
> redundant 
> > with setting IRD and ORD to zero. A modify QP can change 
> IRD and ORD. 
> > Setting up the QP with Inbound RDMA Read Response enabled 
> and ORD zero 
> > would allow a later attempt to modify the QP to increase ORD. My 
> > expectation is that when a QP was set up with Inbound RDMA Read 
> > Response disabled, the RNIC can assume that RDMA Read 
> initiation will 
> > never be allowed and any attempt to allocate ORD depth could be 
> reaction of an 
> > RNIC to a Create QP with Inbound RDMA write and RDMA Read Response 
> > disabled and ORD non-zero or a Modify QP for non-zero ORD 
> when this is 
> > disabled are not described.
> > 
> > It also isn't clear why Inbound RDMA Write and inbound RDMA Read 
> > response share a single enable. They require different 
> resources since 
> > the RDMA Read Response requires ORD depth.
> > They aren't always needed together. An iSCSI target will receive 
> > inbound RDMA Writes and will not receive RDMA Read 
> responses. An iSCSI 
> > From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of 
> > Caitlin Bestler