Subject: RE: [Ipoverib] ipoib-cm multiple connection model

From: Dror Goldenberg <[email-masked]>
Date: Thu, 20 Oct 2005 09:39:26 +0200

After giving it some more thought, I think that there is still a problem.
Please look at the attached pdf.
 
In the first slide, everything happens smoothly, the left peer knows that it
has to accept the connection and the right peer knows that it has to
decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric.
It is retransmitted independently by the CM way after a channel has already
been established. Now the right side can not really tell whether this is a
new channel request or an old "simultaneous connection channel". This may
lead to two channels being opened mistakenly.
 
I don't believe it can be solved by looking at the communication ID. This
number is just being fabricated by the sender of the REQ and is just a
"random number" with respect to what we're trying to determine.
 
-Dror

-----Original Message-----
From: Dror Goldenberg 
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; [email-masked]
Subject: RE: [Ipoverib] ipoib-cm multiple connection model

Hi Vandana,
 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at
the Communication ID it can make those decisions correctly. The question is
whether this value is exposed through common CM APIs. For all other
purposes, the CM doesn't need to expose the Communication ID to the ULP.
However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that
a proper implementation of IPoIB-CM should monitor Communication ID as part
of the simultaneous connection decision.
 
-Dror

-----Original Message-----
From: Vandana Rao [mailto:[email-masked]]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; [email-masked]
Subject: Re: [Ipoverib] ipoib-cm multiple connection model

Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say
below, i.e. have multiple connection establishment MADs between 2 peers. It
is essentially the same as the "channel ID" that you talk about below. Its
just that it is a generic channel ID for any MAD transactions, not just CM
MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:

I think that the current model for multiple connections establishment
between 
two peers works well for 1 connection but is broken for more than one
connection.
Because of cases where CM packets are dropped or travel on different VLs,
they 
might get reordered on the way and observed in a different order than were
originally
sent. This breaks the current model which assumes that the REQs are sent and

processed in order. 
 
I think that we need to think of a model which is robust to CM packets
reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It
helps 
   synchronizing both peers on which is the channel at question now. The
rest
   of the decision (per channel) of whether to accept or reject can be the
same 
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib
<https://www1.ietf.org/mailman/listinfo/ipoverib> 

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vandana Rao <[email-masked]>
Date: Thu, 20 Oct 2005 10:25:01 -0700

Hi Dror,
I think you are confusing the 2 IDs that come into play in this 
setup. The communication ID is something that the CM entity on a node 
understands but the transaction ID is something that the MAD handling 
entity uses (for retransmission, req/resp matchup etc.).
  So, in the second slide, there should not be a retransmission of 
the left peer REQ MAD since when the left peer accepts the 
connection, it will cancel its outstanding REQ request and hence the 
retransmission should not take place.
-Vandana

At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
>After giving it some more thought, I think that there is still a 
>problem. Please look at the attached pdf.
>
>In the first slide, everything happens smoothly, the left peer knows 
>that it has to accept the connection and the right peer knows that 
>it has to decline.
>On the 2nd slide, the left peer REQ transmission is dropped in the 
>fabric. It is retransmitted independently by the CM way after a 
>channel has already been established. Now the right side can not 
>really tell whether this is a new channel request or an old 
>"simultaneous connection channel". This may lead to two channels 
>being opened mistakenly.
>
>I don't believe it can be solved by looking at the communication ID. 
>This number is just being fabricated by the sender of the REQ and is 
>just a "random number" with respect to what we're trying to determine.
>
>-Dror
>
>Thanks for clarifying this. I agree that if IPoiB-CM implementation 
>looks at the Communication ID it can make those decisions correctly. 
>The question is whether this value is exposed through common CM 
>APIs. For all other purposes, the CM doesn't need to expose the 
>Communication ID to the ULP. However, seems like it's not a big deal 
>to expose this value to the ULP.
>Anyway, it worth noting this explanation in the RFC so that it's 
>clear that a proper implementation of IPoIB-CM should monitor 
>Communication ID as part of the simultaneous connection decision.
>
>-Dror
>The purpose of the MAD transaction ID is to allow one to do what you 
>say below, i.e. have multiple connection establishment MADs between 
>2 peers. It is essentially the same as the "channel ID" that you 
>talk about below. Its just that it is a generic channel ID for any 
>MAD transactions, not just CM MADs.
>>I think that the current model for multiple connections 
>>establishment between
>>two peers works well for 1 connection but is broken for more than 
>>one connection.
>>Because of cases where CM packets are dropped or travel on 
>>different VLs, they
>>might get reordered on the way and observed in a different order 
>>than were originally
>>I think that we need to think of a model which is robust to CM 
>>packets reordering
>>which can be one of:
>>- Adding a "channel ID" in the private data. The field goes 0,1,2 
>>and on. It helps
>>    synchronizing both peers on which is the channel at question 
>> now. The rest
>>    of the decision (per channel) of whether to accept or reject 
>> can be the same
>>    as defined in the draft.

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 25 Oct 2005 14:37:38 -0700 (PDT)

Hi Vandana,

I too agree there is no problem here.

I think you are right if the left peer accepts the connection. However,
if the reverse was the case, that is the left peer rejected the connection
since it is found that the local address is numerically larger. It will
do this comparison since its REQ is also outstanding.  The right peer will
not receive any REQ  (it is delayed or yet to be retransmitted). If it
receives the reject it will cancel its request. At a later time it will
receive the delayed/retransmitted request and one connection will be 
established.

Dror, do these two scenarios look right to you?

Vivek

On Thu, 20 Oct 2005, Vandana Rao wrote:

> Hi Dror,
> I think you are confusing the 2 IDs that come into play in this setup. The 
> communication ID is something that the CM entity on a node understands but 
> the transaction ID is something that the MAD handling entity uses (for 
> retransmission, req/resp matchup etc.).
> So, in the second slide, there should not be a retransmission of the left 
> peer REQ MAD since when the left peer accepts the connection, it will cancel 
> its outstanding REQ request and hence the retransmission should not take 
> place.
> -Vandana
>
>> In the first slide, everything happens smoothly, the left peer knows that 
>> it has to accept the connection and the right peer knows that it has to 
>> decline.
>> Thanks for clarifying this. I agree that if IPoiB-CM implementation looks 
>> at the Communication ID it can make those decisions correctly. The question 
>> is whether this value is exposed through common CM APIs. For all other 
>>> - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. 
>>> It helps