Subject: Re: [rddp] MPA issue (3): Rejected Connection bit

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 7 Apr 2004 07:18:54 -0700

At 08:21 PM 3/29/2004, [email-masked] wrote:
>I suggest to the WG that the right thing to do is include this bit
>because it produces better behavior in an important error case for
>cooperative clients where something is misconfigured.  This bit
>allows the RDDP layers to immediately know that a connection setup
>has failed and release/reuse any resources committed to it without
>waiting for a ULP to look at the private data that came back.  It
>also ties the fact that the connection is closing with the
>associated private data unambiguously (both can be immediately
>logged at RDDP level).  Yes, this is optimizing an error case,
>but it seems to be common enough/of sufficient concern to be
>worth spending a bit on.

I agree with this, and would add that by not having the bit, RDDP
violates layering requirements on TCP regarding the FIN. In order
to unambiguously indicate "reject" to the RDDP peer, the TCP FIN
must accompany the returned private data. Otherwise, the
initiating peer must wait for a subsequent close. Also, with the
bit, the connection may remain open for further operation in
streaming mode, if desired.

Here is some proposed text around the reject bit.

The Reject bit provides a means for the responding MPA peer to
reject an incoming RDDP connection. When an incoming connection
is unacceptable to the responding peer, the Reject bit MUST be
set in the reply, along with any ULP-specific private data. When
the Reject bit is true, the connection remains in stream (TCP) mode,
and may be used and/or subsequently closed by either peer.

Additional justifications:

Enhances interoperability by not leaving reject to the upper
layer. This permits ULP-independent rejection, and does not
require ULP private data manipulation.

Simple, non-complicated change to MPA (single bit).

Does not require the receiver to wait for subsequent event,
as current proposal does (the FIN).

Allows receiver to clean up MPA resources immediately
upon rejection, avoiding resource retention if connection
not closed.

Tom.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 7 Apr 2004 10:46:00 -0700

In my view, the state of the connection after an MPA reject is
up the responding ULP (i.e. the one that rejected).

The connection, initially, remains in TCP streaming mode. The ULP
consumer may opt to close the connection, return the connection
to MPA responder mode (before consuming any TCP stream data),
or simply remain in TCP. The decision is not MPA's, nor should it be.

Tom.

At 07:32 PM 4/6/2004, Carrier, John wrote:
>I guess I would like to hear a clarification for how the rejected bit would 
>be used.  
>
>In the startup scenarios described in the MPA draft, when the MPA Responder 
>sends the MPA Reply Frame, the responder 'binds' DDP to MPA so that, after 
>sending the reply frame, the MPA Responder will be ready to receive FPDUs 
>and pass them to DDP.  If the Request Frame is invalid or if the private 
>data is unacceptable, then the ULP should close the connection without 
>sending a Reply Frame.  
>
>Adding the rejected bit enables the ULP to send a Reply Frame without 
>binding DDP to MPA.  The Reply Frame may also contain private data that 
>further describes why the request was rejected.   
>
>If that is the intent, then what should the state of MPA be after sending 
>the Reply Frame with the rejected bit set?  Does it return to 'responder 
>mode' so that the Initiator can resend the MPA Request Frame or does it 
>transition to a new state where it drops any received data while it waits 
>for the ULP to close the connection?
>
>If this has already been discussed on the list, please point me to the thread.
>
>thanks.
>
>--jc
>
>
>John Carrier
>Adaptec
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]On Behalf Of
>> [email-masked]
>> Sent: Monday, March 29, 2004 5:21 PM
>> To: [email-masked]
>> Subject: [rddp] MPA issue (3): Rejected Connection bit
>> 
>> 
>> Moving on to issue (3), the Seoul minutes say:
>> 
>>   (3) Rejected Connection bit. Dissension expressed, this leads to 
>>   interoperability issues because it forces a ULP to 
>> interpret private 
>>   data to figure out that the connection setup failed, and requires
>>   MPA to handle TCP close specially (more detail in mail to list).
>>   Further discussion: 
>> 
>>   Q: If MPA does not do "teardown", why is the bit needed to 
>>    free resources at MPA layer? 
>>   A: Practically speaking, implementations will allocate and free 
>>    resources in MPA (e.g. connection model in kernel), this enables 
>>    layering and better control. 
>>   A: The bit will prevent some timed-wait resource consumption issues,
>>    if used properly (this is about cooperating systems, not malicious
>>    resource consumption attacks).
>> 
>>   Rough consensus in room to include the bit. Take to list.
>> 
>> worth spending a bit on.  That was where the discussion among
>> the subset of us who were in Seoul wound up.
>> 
>> Comments?  I intend to let the technical discussion run on this
>> before trying to call consensus on the list.
>> 
>> Thanks,
>> --David
>> 
>> ----------------------------------------------------
>> David L. Black, Senior Technologist
>> EMC Corporation, 176 South St., Hopkinton, MA  01748
>> +1 (508) 293-7953             FAX: +1 (508) 293-7786
>> [email-masked]        Mobile: +1 (978) 394-7754
>> ----------------------------------------------------
>> 
>> 
>> _______________________________________________
>> rddp mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/rddp
>> 
>
>_______________________________________________
>rddp mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Teisberg, Robert" <[email-masked]>
Date: Wed, 7 Apr 2004 12:44:43 -0500

> 
> The APIs make a distinction between a connection request
> refused by the peer and those that failed for other reasons.

There is room for disagreement whether APIs should conform
to protocols or protocols to APIs.  That's why I'm unconvinced
by citing the authority of API specs.  Especially in light of
your other argument....

> This is an important distinction. If a connection failed for
> "other" (non-peer) reasons, the chance of another session
> request being honored is very low -- but the chance that the
> same session will be accepted at some later time is good.
> 
> If the connection was rejected by the peer, the opposite
> is true.
> 
> Generally the applications themselves do not care about
> exactly what network error killed the connection, but there
> is a very big difference between "cannot communicate
> with X" and "X does not want to talk with you."
> 
> 

I find this much more persuasive.  That it keeps us out of
the philosophical tarpit is a bonus.

Bob

---

From: "Carrier, John" <[email-masked]>
Date: Wed, 7 Apr 2004 11:20:00 -0700

I'll follow up to the email I just sent with the 'state machine' perspective. 
 
I think we need to specify what happens after MPA sends the Reply Frame with the reject bit set.  As I proposed, MPA is 'inactive' but the TCP connection is still open.  We could just as easily say that MPA transitions into 'responder mode' again.  
 
In either case, we need to specify whether MPA is still consuming the TCP data stream after sending the Reply Frame with the reject bit set.  Doing so in no way affects the ULP's control of the connection, but I think makes clear what the ULP can expect from its LLP.

--jc

-----Original Message-----
From: Talpey, Thomas [mailto:[email-masked]]
Sent: Wednesday, April 07, 2004 10:46 AM
To: Carrier, John
Cc: [email-masked]
Subject: RE: [rddp] MPA issue (3): Rejected Connection bit

Tom. 

At 07:32 PM 4/6/2004, Carrier, John wrote: 
>> From: [email-masked] [ mailto:[email-masked]]On Behalf Of 
>> [email-masked]

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 7 Apr 2004 18:34:19 -0700

Sounds good to me. Thanks for the clearer rewording.

Only thing to add is that after the last step, the ULP can
choose to return the MPA to Responder mode, or close the
connection, etc as previously discussed.

Tom.

At 02:08 PM 4/7/2004, Carrier, John wrote:
>Okay, so let me take another shot at clarification:
> 
>The draft shows that MPA is in 'responder mode' while waiting for the MPA 
>Request Frame.  Saying that the connection stays in 'TCP stream mode' after 
>sending the Reject Reply Frame is not a sufficient description of the state 
>change in MPA because it was in 'TCP stream mode' while in 'responder mode'.  
> 
>So, from a state machine perspective:
>* the ULP at the responder puts MPA into 'responder mode' while it waits for 
>the Request Frame from the initiator.  In this state, MPA consumes the TCP 
>data stream.
>* after the responding MPA receives an MPA Request Frame from the MPA 
>initiator, it transitions to a 'request received' state.  In this state, the 
>connection is still in TCP stream mode, although MPA is not expecting any 
>other received data from the initiator (should there be an error if it 
>receives any?).  MPA stays in this state until it receives an indication 
>from the ULP to reply to the initiator. 
>* if the Request Frame is valid and the ULP accepts the connection, MPA will 
>transition to 'MPA mode' after sending the Reply Frame with the rejected bit 
>set to 0.  In this state, MPA at the responder is bound to DDP and is 
>waiting for the first FPDU from the initiator.  'Full MPA mode', where the 
>responder can send FPDUs, doesn't occur until after the responder receives 
>the first FPDU from the initiator.
>* if the Request Frame is invalid or if the ULP rejects the connection, MPA 
>will transition to an 'inactive' state after sending the Reply Frame with 
>the rejected bit set to 1.  In this state, the TCP connection is open, but 
>MPA is no longer consuming any of the TCP data stream; instead any received 
>data goes directly to the ULP.
> 
>--jc
> 
>
>From: [email-masked] [mailto:[email-masked]]On Behalf Of Talpey, 
>Thomas
>Sent: Wednesday, April 07, 2004 7:19 AM
>To: [email-masked]
>Subject: Re: [rddp] MPA issue (3): Rejected Connection bit
>
>
>

---

From: [email-masked]
Date: Mon, 12 Apr 2004 17:52:35 -0600

There is also another reason that reversion to non-RDMA would not be useful for iSCSI. The selection of operation over iSER may have influenced the values that were negotiated for other parameters, e.g. digests and PDU sizes. If the RDMA connection fails, then those parameters may be significantly sub-optimal for operation in stream mode. If the trasition into RDMA connection didn't work, then the iSCSI would just close the connection. It can try the negotiation again without offering RDMA operation.

Regards,
Pat

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Mike
Ko
Sent: Friday, April 09, 2004 3:02 PM
To: Caitlin Bestler
Cc: [email-masked]
Subject: RE: [rddp] MPA issue (3): Rejected Connection bit

The handling of the MPA Request Frame was not considered when the iSER 
draft was written.  But since the decision to enter RDMA mode is made 
during the iSCSI login phase in stream mode, if the subsequent MPA Request 
Frame is invalid, it will be treated as an LLP error and presumably the 
MPA layer will be instructed to tear the connection down.

Mike
Sent by:        [email-masked]
To:     "Carrier, John" <[email-masked]>
cc:     "Caitlin Bestler" <[email-masked]>, [email-masked] 
Subject:        RE: [rddp] MPA issue (3): Rejected Connection bit

Carrier, John said:
> Hi Caitlin,
>
> Again a clarification: MPA does not send a Reply Frame unless instructed
> to do so by the ULP.  If MPA receives an invalid Request Frame, then it
> indicates the error to the ULP but takes no other action of its own.
>
> My interpretation of what you describe below is that if there is an
> invalid Request Frame, there should be no Reply Frame generated.  In 
that
> case, MPA would transition to what I called the 'inactive' state after 
it
> passes the error to the ULP. The result would be that MPA is decoupled
> from the TCP stream.  The ULP would then, presumably, terminate the
> connection.
>
> right?
>

Yes, with the understanding that the ULP may have instructed MPA in 
advance
to terminate the connection. But the decision to terminate the connection
should indeed belong to the ULP.

Applications using the "connect or using RDMA or bust" APIs currently
found in DAT and IT-API would presumably prefer for the TCP connection
to be terminated rather than being forced to do a round-trip.

Those doing iSER/SDP over iWarp style setup might want their connection
back to allow them to continue in non-RDMA mode (even though it is
more likely that they negotiated use of RDMA in streaming mode, so
a failure to enter RDMA mode would probably still result in terminating
the connection -- who wants to deal with a peer that changes their
mind that quickly?)