Subject: RE: [Ipoverib] ipoib-cm multiple connection model

From: "Dror Goldenberg" <[email-masked]>
Date: Wed, 16 Nov 2005 01:04:20 +0200

I think it's OK. We can move forward from my perspective.

> -----Original Message-----
> From: H.K. Jerry Chu [mailto:[email-masked]] 
> Sent: Monday, November 14, 2005 9:34 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> 
> 
> Hi folks,
> 
> Have we reached a concensus the connection setup procedure in 
> the current draft is "good enough", or folks still think it 
> needs to be improved?
> 
> I'd like to see any remaining issue resolved asap so we can 
> start a WG last call (after Vivke revs the draft based on all 
> the comments so far).
> 
> Jerry
> 
> >Date: Thu, 27 Oct 2005 22:47:16 -0700 (PDT)
> >From: Vivek Kashyap <[email-masked]>
> >X-X-Sender: [email-masked]
> >To: Dror Goldenberg <[email-masked]>
> >Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >MIME-Version: 1.0
> >X-Spam-Score: 0.0 (/)
> >X-Scan-Signature: f8ee348dcc4be4a59bc395f7cd6343ad
> >Cc: "[email-masked]'" <[email-masked]>, Vandana Rao 
> ><[email-masked]>
> >X-BeenThere: [email-masked]
> >X-Mailman-Version: 2.1.5
> >List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> >List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>, 
> <mailto:[email-masked]?subject=unsubscribe>
> >List-Post: <mailto:[email-masked]>
> >List-Help: <mailto:[email-masked]?subject=help>
> >List-Subscribe: <https://www1.ietf.org/mailman/listinfo/ipoverib>,
> <mailto:[email-masked]?subject=subscribe>
> >
> >Hi Dror,
> >
> >On Thu, 27 Oct 2005, Dror Goldenberg wrote:
> >
> >> Hi Vivek,
> >>
> >> I agree with the flows that you describe, and with the 
> fact that the 
> >> original requester has always the "second chance" to withdraw its 
> >> request by rejecting the REP. However, in my opinion, 
> giving all this 
> >> freedom to the implementation can lead to interoperability 
> issues and 
> >> poor implementations. If an implementation tries to open n 
> >> connections with a certain peer, it may end up opening 
> more or less 
> >> than n and that might take few iterations and that all 
> depends on the 
> >> CM timing and some other "unrelated" parameters - how can you even 
> >> test it ? I therefore feel that we should provide a robust and 
> >> deterministic specification of how to establish more than one 
> >> connection in the RFC or live with a single connection.
> >
> >I'd like to delve into this a bit more to be sure. As per 
> >your/Vandana's comments, the left peer will cancel the 
> request. Isn't 
> >this is a 'conscious' decision by the ULP? The ULP would then remove 
> >the its internal object that it would associate with this IB 
> >connection. If it was successful in cancelling the REQ then there is 
> >only one connection.
> >
> >If, however, as you pointed out it is unable to, it might 
> receive a REP
> >*if* the remote peer accepts multiple connections. The left 
> peer's ulp 
> >however will not find any object associated with the private 
> data and 
> >the connection request (there will be another one already existing 
> >though) and can use that to indicate rejection.
> >
> >Does the above sound right?
> >
> >Vivek
> >
> >>
> >> BTW, this is my opinion in this case, and if folks feel OK 
> with the 
> >> current draft, then maybe we should move on ?
> >>
> >> Thanks
> >> Dror
> >>
> >>> -----Original Message-----
> >>> From: Vivek Kashyap [mailto:[email-masked]]
> >>> Sent: Wednesday, October 26, 2005 10:33 PM
> >>> To: Dror Goldenberg
> >>> Cc: Vandana Rao; [email-masked]'
> >>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>
> >>>
> >>> On Wed, 26 Oct 2005, Dror Goldenberg wrote:
> >>>
> >>>> Vivek,
> >>>>
> >>>> I agree that the two scenarios you mention here are the 
> interesting 
> >>>> ones. I don't agree about the solution. Let's look at the two 
> >>>> cases:
> >>>>
> >>>> 1) Left peer accepts
> >>>> 	I disagree. You have an assumption that left peer can
> >>>> 	revoke its original connection request - BEFORE transmitting
> >>>> 	the REP that accepts the peer request. I am not sure that
> >>>> 	you can do it with every CM implementation. Even if you can,
> >>>> 	then the HCA can reorder those packets if they are on
> >>>> 	different QPs, and the fabric can reorder them as well if
> >>>> 	they are on different VLs.
> >>>
> >>> Yes, that has the potential for extra connections..however, the 
> >>> following will occur if CM checks don't work.
> >>>
> >>> Left peer accepted. Sent a REP. And of course the REQ was 
> also sent. 
> >>> The right receiver can receive :
> >>>
> >>>  	a. REP first
> >>>
> >>>  	It sets up the connection. Then receives a REQ.
> >>>
> >>>  	i) If it accepts multiple connections from the same peer:
> >>>  		 it will  REP with a new QP (connection).
> >>>
> >>>  		Now, if the left peer decides that it didn't
> >>> need another
> >>>  		connection it terminates the connection.
> >>>
> >>>  		Otherwise, it is legitimate to setup multiple 
> connections
> >>>  		since it is equivalent to a second valid request.
> >>>
> >>>  	ii) If the right peer does not accept multiple connections from 
> >>> the
> >>>  	same peer then it will reject the request.
> >>>
> >>>  	Note that these decisions in part depend on the ipoib component 
> >>> peeking
> >>>  	into the local link address and not on CM.
> >>>
> >>>
> >>>  	b. REQ first
> >>>
> >>>  	Since its own REQ is outstanding, and as per link
> >>> address comparison,
> >>>  	the left peer accepts then this will be rejected.
> >>>
> >>> Vivek
> >>>>
> >>>> 2) Left peer rejects
> >>>> 	I think that this scenario works.
> >>>>
> >>>> Thanks
> >>>> Dror
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Vivek Kashyap [mailto:[email-masked]]
> >>>>> Sent: Tuesday, October 25, 2005 11:38 PM
> >>>>> To: Vandana Rao
> >>>>> Cc: Dror Goldenberg; [email-masked]'
> >>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>
> >>>>>
> >>>>> Hi Vandana,
> >>>>>
> >>>>> I too agree there is no problem here.
> >>>>>
> >>>>> I think you are right if the left peer accepts the connection. 
> >>>>> However, if the reverse was the case, that is the left
> >>> peer rejected
> >>>>> the connection since it is found that the local address is 
> >>>>> numerically larger. It will do this comparison since its
> >>> REQ is also
> >>>>> outstanding.  The right peer will not receive any REQ  (it
> >>> is delayed
> >>>>> or yet to be retransmitted). If it receives the reject it
> >>> will cancel
> >>>>> its request. At a later time it will receive the 
> >>>>> delayed/retransmitted request and one connection will be 
> >>>>> established.
> >>>>>
> >>>>> Dror, do these two scenarios look right to you?
> >>>>>
> >>>>> Vivek
> >>>>>
> >>>>> On Thu, 20 Oct 2005, Vandana Rao wrote:
> >>>>>
> >>>>>> Hi Dror,
> >>>>>> I think you are confusing the 2 IDs that come into play in
> >>>>> this setup.
> >>>>>> The
> >>>>>> communication ID is something that the CM entity on a node
> >>>>> understands but
> >>>>>> the transaction ID is something that the MAD handling
> >>>>> entity uses (for
> >>>>>> retransmission, req/resp matchup etc.).
> >>>>>> So, in the second slide, there should not be a
> >>>>> retransmission of the left
> >>>>>> peer REQ MAD since when the left peer accepts the
> >>>>> connection, it will cancel
> >>>>>> its outstanding REQ request and hence the retransmission
> >>>>> should not take
> >>>>>> place.
> >>>>>> -Vandana
> >>>>>>
> >>>>>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
> >>>>>>> After giving it some more thought, I think that there 
> is still a 
> >>>>>>> problem. Please look at the attached pdf.
> >>>>>>>
> >>>>>>> In the first slide, everything happens smoothly, the left
> >>>>> peer knows
> >>>>>>> that
> >>>>>>> it has to accept the connection and the right peer knows
> >>>>> that it has to
> >>>>>>> decline.
> >>>>>>> On the 2nd slide, the left peer REQ transmission is
> >>>>> dropped in the fabric.
> >>>>>>> It is retransmitted independently by the CM way after a
> >>>>> channel has already
> >>>>>>> been established. Now the right side can not really tell
> >>>>> whether this is a
> >>>>>>> new channel request or an old "simultaneous connection
> >>>>> channel". This may
> >>>>>>> lead to two channels being opened mistakenly.
> >>>>>>>
> >>>>>>> I don't believe it can be solved by looking at the
> >>>>> communication ID.
> >>>>>>> This
> >>>>>>> number is just being fabricated by the sender of the REQ
> >>>>> and is just a
> >>>>>>> "random number" with respect to what we're trying to 
> determine.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Dror Goldenberg
> >>>>>>> Sent: Wednesday, October 19, 2005 9:37 PM
> >>>>>>> To: 'Vandana Rao'; Dror Goldenberg; [email-masked]
> >>>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Vandana,
> >>>>>>>
> >>>>>>> Thanks for clarifying this. I agree that if IPoiB-CM
> >>>>> implementation
> >>>>>>> looks
> >>>>>>> at the Communication ID it can make those decisions
> >>>>> correctly. The question
> >>>>>>> is whether this value is exposed through common CM APIs.
> >>>>> For all other
> >>>>>>> purposes, the CM doesn't need to expose the Communication
> >>>>> ID to the ULP.
> >>>>>>> However, seems like it's not a big deal to expose this
> >>>>> value to the ULP.
> >>>>>>> Anyway, it worth noting this explanation in the RFC so
> >>>>> that it's clear that
> >>>>>>> a proper implementation of IPoIB-CM should monitor
> >>>>> Communication ID as part
> >>>>>>> of the simultaneous connection decision.
> >>>>>>>
> >>>>>>> -Dror
> >>>>>>> -----Original Message-----
> >>>>>>> From: Vandana Rao [mailto:[email-masked]]
> >>>>>>> Sent: Wednesday, October 19, 2005 6:40 PM
> >>>>>>> To: Dror Goldenberg; [email-masked]
> >>>>>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
> >>>>>>>
> >>>>>>> Hi Dror,
> >>>>>>> The purpose of the MAD transaction ID is to allow one to
> >>>>> do what you
> >>>>>>> say
> >>>>>>> below, i.e. have multiple connection establishment MADs
> >>>>> between 2 peers. It
> >>>>>>> is essentially the same as the "channel ID" that you talk
> >>>>> about below. Its
> >>>>>>> just that it is a generic channel ID for any MAD
> >>>>> transactions, not just CM
> >>>>>>> MADs.
> >>>>>>> I do not believe there is any issue with existing IB CM in
> >>>>> this regard.
> >>>>>>> -Vandana
> >>>>>>>
> >>>>>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
> >>>>>>>> I think that the current model for multiple connections 
> >>>>>>>> establishment between two peers works well for 1 
> connection but 
> >>>>>>>> is broken for
> >>>>> more than one
> >>>>>>>> connection.
> >>>>>>>> Because of cases where CM packets are dropped or travel
> >>>>> on different VLs,
> >>>>>>>> they
> >>>>>>>> might get reordered on the way and observed in a
> >>>>> different order than were
> >>>>>>>> originally
> >>>>>>>> sent. This breaks the current model which assumes that
> >>>>> the REQs are sent
> >>>>>>>> and
> >>>>>>>> processed in order.
> >>>>>>>>
> >>>>>>>> I think that we need to think of a model which is 
> robust to CM 
> >>>>>>>> packets reordering which can be one of:
> >>>>>>>> - Adding a "channel ID" in the private data. The field
> >>>>> goes 0,1,2 and on.
> >>>>>>>> It helps
> >>>>>>>>    synchronizing both peers on which is the channel at
> >>>>> question now. The
> >>>>>>>> rest
> >>>>>>>>    of the decision (per channel) of whether to accept or
> >>>>> reject can be the
> >>>>>>>> same
> >>>>>>>>    as defined in the draft.
> >>>>>>>> - Not supporting multiple connections between the same peers.
> >>>>>>>>
> >>>>>>>> -Dror
> >>>>>>>> _______________________________________________
> >>>>>>>> IPoverIB mailing list
> >>>>>>>> [email-masked]
> >>> https://www1.ietf.org/mailman/listinfo/ipoverib
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib
> 

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 15 Nov 2005 15:06:34 -0800

ok..I'll target submitting the updated draft by monday. 

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
Ph: 503 578 3422 T/L: 775 3422

"Dror Goldenberg" <[email-masked]>
Sent by: [email-masked]
11/15/2005 03:04 PM
 
        To:     "H.K. Jerry Chu" <[email-masked]>, 
<[email-masked]>, [email-masked]
        cc:     [email-masked], [email-masked]
        Subject:        RE: [Ipoverib] ipoib-cm multiple connection model