Subject: Re: [Ipoverib] recharter consideration

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 14 Oct 2002 13:33:22 -0700 (PDT)

>> For IPoIB through a non-UD transport, what benefit does RD provide beyond
>> what you can attain through UC or RC?
>> 
>> Regarding demuxing IP datagrams over different QPs, who else (besides
>> Vivek) thinks this is useful enough to be considered?
>
>
>A short comparison:
>
>    QP based demuxing is the smallest delta over IPoIB-UD, followed by
>    UC that adds RDMA Write to QP demultiplexing but drops multicast
>    support. UC's large MTU support is deficient since it is an
>    unreliable mode and hence not of any help in case of lost
>    fragments. RC and RD go beyond UC by providing Large MTU, RDMA
>    Read/Write, Automatic Path Migration, reliability. RD has the
>    restriction that no new message can be transmitted from a given QP
>    unless the previous message has been acknowledged by the peer.
>    Additionally all HCAs are required to support RC but not RD.
>
>
>In my opinion IPoIB-RC is clearly the most useful mode. UC doesn't
>seem useful, QP demux would be a niche use, RD I can't say but am not
>too much inclined towards since it is not supported by all HCAs. I
>guess we could consider modes other than RC as long as there is some
>usefulness since similar steps will be required to setup any of the
>above modes. All require the use of CM to setup a 'connection'. But
>considering all the issues RC is the most promising.

Putting unreliable datagrams through a reliable link layer may not be
a good fit in all cases. E.g. latency sensitive UDP traffic may only
need a best effort delivery and may not want the link-layer in-order
delivery algorithm to get in the way. UDP/IP/UC will enable the use
of large messages to avoid IP fragmentation.

So I think we should consider both UC and RC.

Jerry

>
>Vivek
>
>> 
>> Thanks,
>> 
>> Jerry
>> 
>> >From: Vivek Kashyap <[email-masked]>
>> >Subject: Re: [Ipoverib] recharter consideration
>> >To: [email-masked] (Caitlin Bestler)
>> >Date: Mon, 9 Sep 2002 16:28:41 -0700 (PDT)
>> >Cc: [email-masked]
>> >
>> ><snip..>
>> >> 
>> >> MAC spoofing as a failover does not preserve an existing
>> >> connection. Path migration does. If the goal is to achieve
>> >
>> >Automatic path migration is one advantage that can be derived through
>> >IPoIB-RC. 
>> >
>> >> this sort of connection transparent as to whether an IB or
>> >> IP network is in use then SDP and DAPL would seem to be more
>> >> appropriate.
>> >
>> >If the underlying connection failover is transparent, then the
>> >solution is transparent to the upper layer - whatever it may be
>> >IP/SDP/DAPL. How does it make one protocol more appropriate than the
>> >other? 
>> >
>> >> 
>> >> If the goal is to make a byte or message stream ULP
>> >> connection over an IB network, what benefit is there in
>> >> insisting on tunneling that connection in IP format?
>> >
>> >We are talking about IP over different IB modes and the benefits that
>> >accrue to the IP application such as APM and large MTU. You seem to be
>> >considering replacing IP and using a different ULP. Those are
>> >different discussions.
>> >
>> >> 
>> >> If IPoIB is going to expand its charter, it should be on the
>> >> basis of solving ULP problems, not merely trying to make it
>> >> a universal solution. That becomes an effort in "fixing" IB
>> >> to be more IP like. In my opinion, the end result will be
>> >> unappealing to any target market.
>> >
>> >I for one am interested in a failover of the type mentioned. It really
>> >doesn't matter at which level it is done i.e. at IB level or in IPoIB.
>> >If it can be done at IB level then it doesn't belong in IPoIB but if
>> >it cannot be, then we could consider it here. Does anyone have any
>> >solutions in either space?
>> >
>> >> 
>> >> One legitimate goal that has been cited was supporting a
>> >> larger MTU to allow atomic delivery of larger IP frames, at
>> >> least matching jumbo Ethernet frames. I think that RD and UC
>> >> should be considered for this goal. They meet the objective
>> >> while imposing fewer additional semantics than found in the
>> >> orignal link layer.
>> >
>> >Yes, one gets larger MTU and in the case of RC/RD Automatic path
>> >migration. Any connection/QPN mapping to a particular service reduces
>> >the lookup/scheduling needed in a server.
>> >
>> >I have no objection to considering RD and UC but I don't think they
>> >are in any way 'semantically closer' in context of IPoIB. They are all
>> >'connection' oriented links setup using CM REQ/REP as is the case with
>> >RC. They are point to point exactly as is IPoIB-RC and don't support
>> >multicast. As far as IPoIB is concerned all these modes are
>> >equivalent. If we describe the method for IPoIB over one of these
>> >modes the method can be trivially extended to the other modes. See
>> >section 3.0 in draft-kashyap-ipoib-advanced-capabilities-00.txt.
>> >
>> >Vivek
>> >
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >> Caitlin Bestler
>> >> http://www.asomi.com/CaitlinBestler/
>> >> _______________________________________________
>> >> IPoverIB mailing list
>> >> [email-masked]
>> >> https://www1.ietf.org/mailman/listinfo/ipoverib
>> >> 
>> >
>> >
>> >-- 
>> >Vivek Kashyap
>> >Linux Technology Center, IBM
>> >[email-masked] 
>> >[email-masked] 
>> >503 578 3422 (o)
>> >_______________________________________________
>> >IPoverIB mailing list
>> >[email-masked]
>> >https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Oct 2002 15:08:53 -0700

>So I think we should consider both UC and RC.

I've never seen a reason to support RC for IP-based traffic as the upper 
layers already assume a lossy fabric.  If we were to support any 
improvements, I would hazard that UC provides all of the requisite 
functionality without any additional baggage or transport functionality 
replication.  Keep in mind that IB is essentially loss-less for connected 
traffic sans CRC errors thus why be burdened with RC.

BTW, the retransmission timers are allowed to be fairly large in IB fabrics 
making them unsuitable for real-time traffic such as streaming media unless 
you set them to a small value which may trigger false retransmission 
events.  I certainly don't want to see unreliable traffic flowing over a RC 
link for such workloads.

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 14 Oct 2002 15:08:53 -0700

Mike