Subject: RE: [Ipoverib] recharter consideration

From: "Rosenstock, Hal" <[email-masked]>
Date: Mon, 14 Oct 2002 14:28:40 -0400

Hi again Vivek,

Yes, but if CM doesn't support the transport, I am not sure whether it
matters if the HCA supports the transport as it is not usable (without the
CM support). Wouldn't IPoIB in non UD modes rely on CM (and hence rely on
the CM's ClassPortInfo) ?

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, October 14, 2002 2:32 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Hal,
	
    I think this is a software module only indicator. So these flags
    indicate the capabilities of the supported CM(a software module)
    but the specification otherwise requires all HCAs to support the
    Reliable connected mode.

Vivek

> 
> Hi Vivek,
> 
> It looks like RC support in the CM is optional:
> 
> 1.0.a vol 1 p.821
> Table 254 Communication Management ClassPortInfo:CapabilityMask
> 9 IsReliableConnectionCapable Reliable Connections are supported
> 10 IsReliableDatagramCapable Reliable Datagrams are supported
> 
> While this changes some of the details of what you wrote, I don't
> think it materially affects your conclusions.
> 
> -- Hal
> 
> -----Original Message-----
> From: Vivek Kashyap [mailto:[email-masked]]
> Sent: Monday, October 14, 2002 1:01 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [Ipoverib] recharter consideration
> 
> 
> > 
> > Hello,
> > 
> > Trying to close some loops due to lost email.
> > 
> > On the failover of IP addresses to different CAs, a pure link layer
> > failover solution doesn't belong to IPoIB charter. But an IB failover
> > mechanism specifically for IP is architectually questionable.
> > Can we get a better idea what the proposed solution might look like and
> > what benefit it provides?
> 
> I agree, it will be useful to get an outline of a possible solution.
> 
> > 
> > For IPoIB through a non-UD transport, what benefit does RD provide
beyond
> > what you can attain through UC or RC?
> > 
> > Regarding demuxing IP datagrams over different QPs, who else (besides
> > Vivek) thinks this is useful enough to be considered?
> 
> 
> A short comparison:
> 
>     QP based demuxing is the smallest delta over IPoIB-UD, followed by
>     UC that adds RDMA Write to QP demultiplexing but drops multicast
>     support. UC's large MTU support is deficient since it is an
>     unreliable mode and hence not of any help in case of lost
>     fragments. RC and RD go beyond UC by providing Large MTU, RDMA
>     Read/Write, Automatic Path Migration, reliability. RD has the
>     restriction that no new message can be transmitted from a given QP
>     unless the previous message has been acknowledged by the peer.
>     Additionally all HCAs are required to support RC but not RD.
> 
> 
> In my opinion IPoIB-RC is clearly the most useful mode. UC doesn't
> seem useful, QP demux would be a niche use, RD I can't say but am not
> too much inclined towards since it is not supported by all HCAs. I
> guess we could consider modes other than RC as long as there is some
> usefulness since similar steps will be required to setup any of the
> above modes. All require the use of CM to setup a 'connection'. But
> considering all the issues RC is the most promising.
> 
> Vivek
> 
> > 
> > Thanks,
> > 
> > Jerry
> > 
> > >From: Vivek Kashyap <[email-masked]>
> > >Subject: Re: [Ipoverib] recharter consideration
> > >To: [email-masked] (Caitlin Bestler)
> > >Date: Mon, 9 Sep 2002 16:28:41 -0700 (PDT)
> > >Cc: [email-masked]
> > >
> > ><snip..>
> > >> 
> > >> MAC spoofing as a failover does not preserve an existing
> > >> connection. Path migration does. If the goal is to achieve
> > >
> > >Automatic path migration is one advantage that can be derived through
> > >IPoIB-RC. 
> > >
> > >> this sort of connection transparent as to whether an IB or
> > >> IP network is in use then SDP and DAPL would seem to be more
> > >> appropriate.
> > >
> > >If the underlying connection failover is transparent, then the
> > >solution is transparent to the upper layer - whatever it may be
> > >IP/SDP/DAPL. How does it make one protocol more appropriate than the
> > >other? 
> > >
> > >> 
> > >> If the goal is to make a byte or message stream ULP
> > >> connection over an IB network, what benefit is there in
> > >> insisting on tunneling that connection in IP format?
> > >
> > >We are talking about IP over different IB modes and the benefits that
> > >accrue to the IP application such as APM and large MTU. You seem to be
> > >considering replacing IP and using a different ULP. Those are
> > >different discussions.
> > >
> > >> 
> > >> If IPoIB is going to expand its charter, it should be on the
> > >> basis of solving ULP problems, not merely trying to make it
> > >> a universal solution. That becomes an effort in "fixing" IB
> > >> to be more IP like. In my opinion, the end result will be
> > >> unappealing to any target market.
> > >
> > >I for one am interested in a failover of the type mentioned. It really
> > >doesn't matter at which level it is done i.e. at IB level or in IPoIB.
> > >If it can be done at IB level then it doesn't belong in IPoIB but if
> > >it cannot be, then we could consider it here. Does anyone have any
> > >solutions in either space?
> > >
> > >> 
> > >> One legitimate goal that has been cited was supporting a
> > >> larger MTU to allow atomic delivery of larger IP frames, at
> > >> least matching jumbo Ethernet frames. I think that RD and UC
> > >> should be considered for this goal. They meet the objective
> > >> while imposing fewer additional semantics than found in the
> > >> orignal link layer.
> > >
> > >Yes, one gets larger MTU and in the case of RC/RD Automatic path
> > >migration. Any connection/QPN mapping to a particular service reduces
> > >the lookup/scheduling needed in a server.
> > >
> > >I have no objection to considering RD and UC but I don't think they
> > >are in any way 'semantically closer' in context of IPoIB. They are all
> > >'connection' oriented links setup using CM REQ/REP as is the case with
> > >RC. They are point to point exactly as is IPoIB-RC and don't support
> > >multicast. As far as IPoIB is concerned all these modes are
> > >equivalent. If we describe the method for IPoIB over one of these
> > >modes the method can be trivially extended to the other modes. See
> > >section 3.0 in draft-kashyap-ipoib-advanced-capabilities-00.txt.
> > >
> > >Vivek
> > >
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> Caitlin Bestler
> > >> http://www.asomi.com/CaitlinBestler/
> > >> _______________________________________________
> > >> IPoverIB mailing list
> > >> [email-masked]
> > >> https://www1.ietf.org/mailman/listinfo/ipoverib
> > >> 
> > >
> > >
> > >-- 
> > >Vivek Kashyap
> > >Linux Technology Center, IBM
> > >[email-masked] 
> > >[email-masked] 
> > >503 578 3422 (o)
> > >_______________________________________________
> > >IPoverIB mailing list
> > >[email-masked]
> > >https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> 
> 
> -- 
> Vivek Kashyap
> Linux Technology Center, IBM
> [email-masked] 
> [email-masked] 
> 503 578 3422 (o)
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib
> 

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 14 Oct 2002 11:52:54 -0700 (PDT)

Yes, that is true. CM is needed for RC or any of the non-UD modes. 

I only mean that one can add the CM software module if the one
available doesn't support RC mode since the hardware will always
support it. But since RD is not required to be supported just getting
the right software is not enough - the right kind of HCA must also be
acquired. 

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 14 Oct 2002 11:52:54 -0700 (PDT)

Vivek