Subject: Re: [Ipoverib] recharter consideration

From: "Vivek Kashyap" <[email-masked]>
Date: Wed, 16 Oct 2002 15:50:59 -0700

Mike,

We can consider both RC and UC. Other than their intrinsic link layer
qualities or modes both RC and UC look the same to the IP layer - similar
setup and use.

[ UC will not do well in face of a loss since the receiver just drops the
fragment and the rest of the message silently. It might be better to use
smaller MTU based UD based transfer since a loss of small data  can be
overcome by the application.  It is also better to use smaller packets  if
low latency is desired. Also if suitable buffering is done by the
application RC can easily retransmit the missing fragment in 1 RTT
providing a better service than UC which will drop all data. The data we
are talking about is based on large MTUs - 32 K of IP MTU implies 16
fragments in a single UC link for a 2K link. With all this I still don't
have an argument with using UC for streaming data if found appropriate in
certain situations but UC is not suitable for all applications. ]

Vivek

--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Michael Krause                                                                                                   
                      <[email-masked]        To:       "H.K. Jerry Chu" <[email-masked]>, [email-masked]               
                      m>                       cc:       [email-masked]                                                             
                      Sent by:                 Subject:  Re: [Ipoverib] recharter consideration                                        
                      ipoverib-admin@ie                                                                                                
                      tf.org                                                                                                           

                      10/14/2002 03:08                                                                                                 
                      PM                                                                                                               

At 01:33 PM 10/14/2002 -0700, H.K. Jerry Chu wrote:
> >> For IPoIB through a non-UD transport, what benefit does RD provide
beyond
> >> what you can attain through UC or RC?
> >>
> >> Regarding demuxing IP datagrams over different QPs, who else (besides
> >> Vivek) thinks this is useful enough to be considered?
> >
> >
> >A short comparison:
> >
> >    QP based demuxing is the smallest delta over IPoIB-UD, followed by
> >    UC that adds RDMA Write to QP demultiplexing but drops multicast
> >    support. UC's large MTU support is deficient since it is an
> >    unreliable mode and hence not of any help in case of lost
> >    fragments. RC and RD go beyond UC by providing Large MTU, RDMA
> >    Read/Write, Automatic Path Migration, reliability. RD has the
> >    restriction that no new message can be transmitted from a given QP
> >    unless the previous message has been acknowledged by the peer.
> >    Additionally all HCAs are required to support RC but not RD.
> >
> >
> >In my opinion IPoIB-RC is clearly the most useful mode. UC doesn't
> >seem useful, QP demux would be a niche use, RD I can't say but am not
> >too much inclined towards since it is not supported by all HCAs. I
> >guess we could consider modes other than RC as long as there is some
> >usefulness since similar steps will be required to setup any of the
> >above modes. All require the use of CM to setup a 'connection'. But
> >considering all the issues RC is the most promising.
>
>Putting unreliable datagrams through a reliable link layer may not be
>a good fit in all cases. E.g. latency sensitive UDP traffic may only
>need a best effort delivery and may not want the link-layer in-order
>delivery algorithm to get in the way. UDP/IP/UC will enable the use
>of large messages to avoid IP fragmentation.
>
>So I think we should consider both UC and RC.

I've never seen a reason to support RC for IP-based traffic as the upper
layers already assume a lossy fabric.  If we were to support any
improvements, I would hazard that UC provides all of the requisite
functionality without any additional baggage or transport functionality
replication.  Keep in mind that IB is essentially loss-less for connected
traffic sans CRC errors thus why be burdened with RC.

BTW, the retransmission timers are allowed to be fairly large in IB fabrics

making them unsuitable for real-time traffic such as streaming media unless

you set them to a small value which may trigger false retransmission
events.  I certainly don't want to see unreliable traffic flowing over a RC

link for such workloads.

Mike

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Thu, 17 Oct 2002 06:56:30 -0700

>certain situations but UC is not suitable for all applications. ]

Could not more strongly disagree - UC is all that is required:

- IB is not the Internet, i.e. it is a controlled / well-understood fabric 
within a data center

- IB is a strongly ordered, essentially loss-less fabric

- IB only drops packets due to a CRC error

- CRC rates should be in the 10^-12 to 10^-15 range with most engineers 
believing 10^-15 will be common for deployments.

- Extremely large percentage of packets will be less than 2KB for a variety 
of workloads.

- Use of UC eliminates ACK packets being exchanged thus reducing fabric 
workload and increasing overall fabric efficiency.

- IB components have thin buffering and depending upon the workload and 
number of QP operating over a given CA, they may also have thin memory 
resources to accommodate retransmission.

- It is simpler and easier for a TCA to implement UC rather than RC as it 
does not have to maintain much in terms of state or resources and can more 
easily flow through to its secondary fabric as a result.

Conclusion: UC will work just fine as RC will provide no value for IP 
workloads and in fact, will result in lost fabric efficiency and 
potentially increased probability of congestion events.

Mike

---

From: Michael Krause <[email-masked]>
Date: Thu, 17 Oct 2002 06:56:30 -0700

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Oct 2002 09:59:30 -0500

>Could not more strongly disagree - UC is all that is

I agree, precisely for the reasons cited (but not quoted).
UC provides the only required benefit: ending PMTU
dependencies, but does so with the *minimal* extra
complexity in the network exchanges.

Implicit in some of the arguments for RC is the assumption
that RC will be better supported than UC. Even if this were
true, the limitations of *some* local interfaces should not
be a factor dictating decisions on a wire protocol.

Clearly UC is the *simplest* solution to solve the problem.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 17 Oct 2002 09:59:30 -0500

Caitlin Bestler

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Oct 2002 12:26:47 +0300

Mike,

What about the benefit of having APM support in RC ?

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Michael Krause
Sent: Thursday, October 17, 2002 4:57 PM
To: Vivek Kashyap
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Fri, 18 Oct 2002 12:26:47 +0300

Mike,

Yaron

-----Original Message-----

Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 18 Oct 2002 10:44:39 -0500

>What about the benefit of having APM support in RC ?
Heroic efforts to deliver an IP datagram at a virtual
link layer is inconsistent with the expectations of the
IP layer.

What if the IP datagrams being delivered over this "link"
are RTP/UDP packets? The application expects a predictable,
relatively consistent, delay. UC and UD are consistent with
that expectation.

Enhancements to IPoverIB should be consistent with the
mission of carrying IP datagrams. Trying to roll in APM
support strikes me as trying to design an alternate
transport for InfiniBand hardware, which in my opinion
should remain outside the scope of the charter.

Caitlin Bestler

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 18 Oct 2002 10:44:39 -0500

Caitlin Bestler

---

From: Michael Krause <[email-masked]>
Date: Fri, 18 Oct 2002 10:34:32 -0700

>What about the benefit of having APM support in RC ?

APM should work just fine for UC.

Mike

---

From: Michael Krause <[email-masked]>
Date: Fri, 18 Oct 2002 10:34:32 -0700

Mike

---

From: Gunneswara Marripudi <[email-masked]>
Date: Fri, 18 Oct 2002 20:52:16 -0700

In the ipoib-link-multicast draft dated 6/2002, available
at the IETF web site, in Sec8.0/pg8, an example was given
showing the P_Key as 8 (0x0008); i.e. the membership is
limited. But a limited member can't accept packets from another
limited member.

Nowhere in the draft, do the authors ask for a P_Key with
full membership; i.e. with the high order bit set to 1.
I assume it is implied.

For clarification purposes, it may be better to change
the example and add a statement to the effect that P_Key shall
have a full-membership.

-gunna

---

From: Gunneswara Marripudi <[email-masked]>
Date: Fri, 18 Oct 2002 20:52:16 -0700

-gunna