Subject: RE: [rddp] Error code returns for DDP tagged buffer messages

From: "Shah, Hemal" <[email-masked]>
Date: Tue, 27 Apr 2004 08:25:23 -0700

Pat,

See my comments below embedded between <hvs> and </hvs>.

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, April 26, 2004 4:52 PM
To: Shah, Hemal; [email-masked]; [email-masked]
Subject: RE: [rddp] Error code returns for DDP tagged buffer messages

Hemal,

Your responses seem to depend on assumptions about implementation.

Depending implementation, the validity check may allow the
implementation to see that the STag is valid but not associated with
this stream or it may not see anything that distinguishes the two cases.
Also, if the peer has tried to access an STag it shouldn't have access
to, I can't think of any reason to tell the peer that the STag is
possibly valid on another stream. Doing so doesn't convey any useful
information to an honest peer and it potentially reveals informtion to
an attacker.

Therefore, STag not associated with DDP Stream should be returned.
<hvs>
I'm little bit confused here. In my email, I exactly recommended that
the above error should be returned in the case Barry described. It seems
like we are in agreement here. Please clarify.
</hvs>

On the MSN errors, your answer presumes there is a fixed queue size
which there may or may not be. There also isn't a value in
distinguishing between these two errors. A message arrived and there was
no buffer.
<hvs>
I am not assuming a fixed queue size. At any give point in time, a queue
can have QLEN size which can change dynamically based on the
modifications to the queue size. For the errors, the queue size seen at
the message arrival time should be used.
</hvs> 

Also, nothing in the draft makes these distinctions between the errors.
In each case the distinction is not particularly useful and I think we
should consolidate to one error for STag not valid on the stream and one
error for no buffer for the MSN.

<hvs>
I believe that the distinction can be useful for the debugging.
</hvs>

Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Shah,
Hemal
Sent: Monday, April 26, 2004 12:25 PM
To: Barry Reinhold; RDDP
Subject: RE: [rddp] Error code returns for DDP tagged buffer messages

Barry,

See my comments below embedded between <hvs> and </hvs>.

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Barry Reinhold
Sent: Friday, April 23, 2004 12:22 PM
To: RDDP
Subject: [rddp] Error code returns for DDP tagged buffer messages

Clause 9.2 DDP Error Numbers provides the error codes "Invalid STag" and
"STag not associated with DDP Stream", but the language of the first
paragraph is not clear enough for me to understand the expectation for
the following:

<hvs>
For an "Invalid STag" memory access is not allowed at all. "STag not
associated with DDP Stream" means that the STag is valid but is not
associated with the DDP Stream on which the DDP Segment was received. In
other words, memory access using the STag is not allowed on this DDP
Stream. 
</hvs>

Is it the intent to require that a receiver delineate between "Invalid
STag" and "STag not associated with DDP Stream"? That is, if STag 10 was
valid on stream A but not on stream B, MUST the receiver return "STag
not associated with DDP Stream?" if a DDP Segment is received on stream
B with STag 10, or is "Invalid STag" acceptable.
<hvs> No. The receiver MUST return "STag not associated with DDP Stream"
in this case. </hvs>

This question also applies to delineating between "Invalid MSN - no
buffer available", and "Invalid MSN - MSN range is not valid". However,
this is a bit more difficult conceptually as Clause 9.1, item 5 states:

       The legal range is defined as being between the MSN value
assigned to
       the first available buffer for a specific QN and the MSN value 
       assigned to the last available buffer for a specific QN.

Which appears to tie "invalid MSN pretty closely to no buffer available.
<hvs>
Let us say QN has a queue length QLEN. So, at any given point [MSN1,
MSN2] represents the MSN range of the entries of the queue. Here,
MSN2-MSN1 = QLEN-1 (in modulo arithmetic). Let us say the number of
buffers posted on the queue at any given time is N (0 <= N <= QLEN).

So, for N > 0, if a MSN is not in [MSN1,MSN1+N-1] and MSN1 <= MSN <=
MSN2, then it is the condition "Invalid MSN - no buffer available". For
N=0, if MSN is in [MSN1, MSN2], then it is the condition ""Invalid MSN -
no buffer available".

If MSN is not in [MSN1,MSN2], then it is "Invalid MSN - MSN range not
valid" condition.
</hvs>  

Barry Reinhold
Lamprey Networks
[email-masked]
(603) 868-8411

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 27 Apr 2004 19:36:22 +0300

> the message arrival time should be used.

You are assuming that there is a queue that is specific to the stream,
and that it's size is easily determined.

That is actually an excellent design. Which is why I was opposed
to the definition of SRQs in the RDMAC iWarp verbs.

If the local interface implements an RDMAC style SRQ it is
obligated to allocate over the nominal per-stream limits if
there are buffers available. Such a constraint all but forces
a more complex SRQ design that is not anything as simple
as an adjustable ring buffer.

With that implementation you *might* be able to check the
per-stream quota whenever a buffer is not available. If the
new MSN minus the stream's base MSN would exceed the
stream's quota (that is, if the buffer had been accepted it
would have raised the per-stream over-limit alarm) then
the implementation could return the "Invalid MSN" error.
But if it is within the limit, meaning the remote peer is
probably not the party responsible for buffer exhaustion,
an "No Available Buffer" error is far more accurate.

However, in order to *reliably* distinguish between the
two the per-stream quota must be available at the same
layer that generates the error. It is entirely plausible
to have an implementation where allocations are
made from the SRQ on the RNIC, but the alarms
are detected by the driver.

My preference for deterministic teardown of streams
that exceed their quota has been previously noted.
But given that the consensus favors allowing asynchronous
detection of this condition (with only the caveat that the
application not be forced into that asynchronous loop)
we have to recognize that valid SRQ implementations
may not always be able to distinguish between the two
errors.

I do not believe we need to disrupt implementations
by running around and renumbering the error conditions.
But we should note that when multiple error returns are
all true then the implementation is free to issue any of
them.  There is no "correct" choice among the error
codes that are true.

The only exception is implied by the DDP ordering rules.
If there is a DDP layer error, then it MUST take precedence
over an RDMAP layer error. After all, evaluating the RDMAP
packet before validating the DDP packet would have been
speculative. Technically, if there is a DDP layer error then
the RDMAP layer should never have gotten the packet
for handling.

> I believe that the distinction can be useful for the debugging.

I agree.

When the receiver knows that the MSN is invalid it SHOULD provide
that diagnostic information. This should be simple with RQs and with
many SRQ designs. And there is a very big difference between "you
sent too many messages" and "I ran out of buffers".

--
Caitlin Bestler
http://asomi.com/

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 27 Apr 2004 19:36:22 +0300

I agree.

--
Caitlin Bestler
http://asomi.com/