Subject: RE: [rddp] Adaption Indication for TCP

From: [email-masked]
Date: Mon, 9 Dec 2002 12:43:05 -0500

> David Black wrote:
> > (4) Assurance of no confusion about presence of framing.
> > 
> > The "gold standard" here is the SCTP adaption indication -
> > if the two ends of the connection don't agree on the usage
> > of DDP or RDMA, the result is no connection.  At the moment,
> > MPA depends on the ULP to get this right, including when to
> > turn MPA on after a connection has been established, so every
> > time a new ULP is developed on MPA, there's a new opportunity
> > to get this wrong.  Asserting that MPA is a ULP to TCP (and hence
> > nothing needs to be done here) is unlikely to fly unless one
> > is prepared to abandon the current link between markers and
> > PDU alignment at TCP senders.
> 
> 
> Personally I find the idea of depending on each ULP to get
> adoption negotiation right less troubling that relying on each
> ULP to get the buffer ownership model correct.  However I will
> save that for a separate thread.
> 
> The current thinking is to allow each ULP to negotiate use
> of DDP and start using DDP at an agreed upon point within
> an already established connection.  The advantage of this
> is that it allows pre-existing ULPs to negotiate an upgrade
> to DDP mode without causing any backward incompatability.
> If either end doesn't understand DDP, then the negotiation
> defaults to non-DDP mode with no ill effects.  The down
> side of this is that the negotiation must necessarily
> by ULP specific.  Therefore each ULP must solve this problem
> for itself.

One could also use an adaption indication class of mechanism as
a safeguard - if it's not provided by the other side, any attempt
to go into DDP mode fails (enforced by the DDP protocol on the side
that wants to go into DDP mode).  This check is somewhat more important
when the connection is supposed to go into DDP mode by default
on establishment than when this is negotiated by the ULP, as one
would hope that by the time a ULP gets far enough into a negotiation
to think about turning DDP on, the ULP would be reasonably certain
that it is talking to its intended peer ULP on the other side (
but bugs happen ...).

So, the request for additional assurance does not require that the
current thinking be discarded.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 9 Dec 2002 14:12:13 -0500

David Black wrote:
> that wants to go into DDP mode).

Not sure I know what "an adaption indication class of mechanism" is.
You are perhaps suggesting an in band double check for what the
ULP has negotiated.  For instance:

  A and B negotiate use of DDP.  MPA or equivalent layer does following
  transition to DDP mode.

  A -> B:   X       -- X is randomly selected value sent from A to B
  B -> A:   F(X)    -- F is non-trivial function defined by MPA
  A -> B:   G(F(X)) -- G is non-trivial function (possible same as F)

      A transitions to framed mode transmit on receipt of F(X)
      B transitions to framed mode transmit on receipt of G(F(X))

  Seems pretty unlikely that either end will transition to MPA mode
  without the knowledge and cooperation of the other.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 9 Dec 2002 19:40:14 -0500

> That works, although I was thinking of something else to be done
> immediately on connection setup - exchange SCTP-like adaption
> indications meaning "I can do MPA" as the first bytes on the
> connection (details of formatting this are TBD) without (necessarily)
> going into MPA mode at that point.  

I don't understand.  If the remote end implements a version of the ULP
that has never heard of RDDP, 
won't the unrecognized first bytes that are supposed to mean
"I can do MPA" simply cause an error and trash
the connection?  This is not the desired behavior of falling back
to non-DDP flavor of the ULP.  Maybe I am missing something.

> The ULP can subsequently
> decide when to put the connection into MPA mode, but if the adaption
> indications have not been sent and received, the local MPA 
> implementation
> is required to reject any attempt to put the connection into MPA mode.
> 
> Thanks,
> --David

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 9 Dec 2002 19:45:25 -0500

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Monday, December 09, 2002 6:07 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: [rddp] Adaption Indication for TCP
>
> A confirming message as the first message to be received in
> MPA mode makes a lot of sense. Particularly if packing is
> allowed and therefore it does not automatically waste a
> distinct transmission.

I thought the objective was to avoid sending anything in 
MPA mode without some assurance the remote end
was willing and able, in which case the confirming message
comes too late.

But I admit I am too vague on what the objective really
is.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Tue, 10 Dec 2002 10:35:58 -0500

> to non-DDP flavor of the ULP.  Maybe I am missing something.

The MPA-enabled sender observes the connection closing and retries
without the extra first bytes.  It's not the cleanest solution, but
we're not exactly starting from a clean slate.

> was willing and able, in which case the confirming message

That gets to about the same place.  In the failure case where
the sender mistakenly believes that the receiver supports MPA,
the result is that the first message in MPA mode should come out
as gibberish on the far side, and when the sender fails to receive
a confirming message, the connection probably has to be closed -
it could be retried without going into MPA mode.  

> But I admit I am too vague on what the objective really

There isn't a hard and fast objective - the originally expressed
concern is that if sender and receiver don't agree on whether
the bytestream is framed, the receiver may do some rather unintended
things.  The above approaches help to address this by causing
"fail fast" behavior - that's not as clean as the SCTP adaption
indication solution (which fails immediately with a clear indication
of why), but something in this area may be sufficient.  I can't
say with absolute certainty what will be acceptable, but with
my WG chair hat on, exploring these sorts of "fail fast" mechanisms
appears to be productive.  Something to be looked at in the
details here is how to increase the likelihood that the
"I'm doing MPA" message really comes out as gibberish at a
non-MPA receiver in a fashion that does not cause any bad
ULP side effects.

Thanks,
--David

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 10 Dec 2002 09:31:48 -0800

Since the use of DDP/RDMA will not be transparent from a ULP perspective,
I'm also confused as to the objective of this thread.  A ULP must know,
through well known ports or a port mapping service, how to find the intended
endpoint.  It seems logical to me that if a ULP supports two separate
transfer methods, it will be know by two well known ports or port mapping
entries.

Can someone site examples of ULPs that dynamically switch transfer methods
over the same TCP port?

Roy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 7:36 AM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Thanks,
--David

---

From: [email-masked]
Date: Tue, 10 Dec 2002 14:10:01 -0700

Roy,

Some ULPs start a connection with a negotiation where they agree on how they will operate. For such ULPs, an RDMA capable instance of the ULP (that is one that knows how to run on top of RDMA) may initially connect to a peer with RDMA turned off. During the negotiation, it can ask the peer to operate over RDMA and if the peer agrees they can both transition into operation over RDMA. These mechanisms can allow for the offer of a new capability and graceful continued operation if the peer does not understand the offer. 

For instance, iSCSI has an operational negotiation phase before it enters full feature phase. RDMA could be negotiated during that phase and then the transition could occur if both agreed. The iSCSI negotiation was already constructed to deal gracefully with receiving an offer of a parameter name that the peer doesn't understand. The peer responds to that parameter with NotUnderstood - so the RDMA capable peer will know that its offer was not accepted and it can continue to negotiate non-RDMA operation.

Some other ULPs use fixed format messages for negotiation in which case a reserved bit could be set to request RDMA operation. If the peer doesn't support RDMA, it would ignore the reserved bit and return it with a zero value. 

I don't think "dynamically switch transfer methods" applies to this. It is a single switch at a defined time agreed upon time with a specified sequence of events. "Dynamically" implies switching while in full operation and perhaps the capability of going back and forth which is much harder.

For ULPs that support a negotiation phase, this seems better then creating a new set of well known port names and it means that existing discovery mechanisms don't need to be modified.

Pat

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 9:32 AM
To: [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Roy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 7:36 AM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Thanks,
--David

---

From: "Larsen, Roy K" <[email-masked]>
Date: Tue, 10 Dec 2002 13:37:16 -0800

Pat,

You've described end-to-end protocols where both sides have a mutual
understanding of the semantics of a negotiation.  I have absolutely no
argument with this approach.  In fact, I believe a formal adaptation-like
end-to-end protocol for MPA could resolve a lot of issues that have been
raised such as marker negotiation, graceful transition from streaming to
framing formats, etc.

However, this thread was not referring to such mutual understanding.  The
approaches I read suggested that when one side closes due to confusion that
the connection is retried on the same port in a different dialect.  This
strikes me as rather amateurish and unworthy of an IETF form.

Again, I was asking if anyone can provide any concrete example where an
application does not deterministically know what dialect the target endpoint
will be speaking.  If there are no precedents, why would we try to solve a
problem that doesn't exist? 

Roy 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 1:10 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Roy,

Some ULPs start a connection with a negotiation where they agree on how they
will operate. For such ULPs, an RDMA capable instance of the ULP (that is
one that knows how to run on top of RDMA) may initially connect to a peer
with RDMA turned off. During the negotiation, it can ask the peer to operate
over RDMA and if the peer agrees they can both transition into operation
over RDMA. These mechanisms can allow for the offer of a new capability and
graceful continued operation if the peer does not understand the offer. 

For instance, iSCSI has an operational negotiation phase before it enters
full feature phase. RDMA could be negotiated during that phase and then the
transition could occur if both agreed. The iSCSI negotiation was already
constructed to deal gracefully with receiving an offer of a parameter name
that the peer doesn't understand. The peer responds to that parameter with
NotUnderstood - so the RDMA capable peer will know that its offer was not
accepted and it can continue to negotiate non-RDMA operation.

Some other ULPs use fixed format messages for negotiation in which case a
reserved bit could be set to request RDMA operation. If the peer doesn't
support RDMA, it would ignore the reserved bit and return it with a zero
value. 

I don't think "dynamically switch transfer methods" applies to this. It is a
single switch at a defined time agreed upon time with a specified sequence
of events. "Dynamically" implies switching while in full operation and
perhaps the capability of going back and forth which is much harder.

For ULPs that support a negotiation phase, this seems better then creating a
new set of well known port names and it means that existing discovery
mechanisms don't need to be modified.

Pat

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 9:32 AM
To: [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Roy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, December 10, 2002 7:36 AM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Adaption Indication for TCP

Thanks,
--David

---

From: "Vadim Makhervaks" <[email-masked]>
Date: Wed, 11 Dec 2002 15:30:10 +0200

What about out-of-order data placement of DDP? How RNIC should behave if
the segment carrying the "confirming message" has been lost? Should it keep
reassembling till this segment is received?

I think that a full-handshake negotiation and a separate negotiation stage
is  cleaner solution  - Can be implemented in software using regular LLP
connection without impact on RNIC.

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |           Sent by:         |
|         |           [email-masked]|
|         |           rg               |
|         |                            |
|         |                            |
|         |           10/12/02 01:07 AM|
|         |                            |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       [email-masked]                                                                                                 |
  |       cc:       [email-masked]                                                                                           |
  |       Subject:  RE: [rddp] Adaption Indication for TCP                                                                        |
  |                                                                                                                               |
  |                                                                                                                               |
  >-------------------------------------------------------------------------------------------------------------------------------|

On 12/9/02, [email-masked] wrote:

>
>That works, although I was thinking of something else to
>be done immediately on connection setup - exchange
>SCTP-like adaption indications meaning "I can do MPA" as
>the first bytes on the connection (details of formatting
>this are TBD) without (necessarily) going into MPA mode at
>that point.  The ULP can subsequently decide when to put
>the connection into MPA mode, but if the adaption
>implementation is required to reject any attempt to put
>the connection into MPA mode.
>

Considering that the first message sent will typically be
untagged, and not necessarily full sized, the chance of a
first "mode confirming message" being 'free' is good.

And even if a confirming message uses an extra transmission,
there is no round-trip confirmation required. It would not
have an adverse impact on getting to work. Requiring a MPA
layer ack for entry would not be a good.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp