Subject: Re: [Ipoverib] recharter consideration

From: "H.K. Jerry Chu" <[email-masked]>
Date: Fri, 4 Oct 2002 17:48:45 -0700 (PDT)

Hello,

Trying to close some loops due to lost email.

On the failover of IP addresses to different CAs, a pure link layer
failover solution doesn't belong to IPoIB charter. But an IB failover
mechanism specifically for IP is architectually questionable.
Can we get a better idea what the proposed solution might look like and
what benefit it provides?

For IPoIB through a non-UD transport, what benefit does RD provide beyond
what you can attain through UC or RC?

Regarding demuxing IP datagrams over different QPs, who else (besides
Vivek) thinks this is useful enough to be considered?

Thanks,

Jerry

>From: Vivek Kashyap <[email-masked]>
>Subject: Re: [Ipoverib] recharter consideration
>To: [email-masked] (Caitlin Bestler)
>Date: Mon, 9 Sep 2002 16:28:41 -0700 (PDT)
>Cc: [email-masked]
>
><snip..>
>> 
>> MAC spoofing as a failover does not preserve an existing
>> connection. Path migration does. If the goal is to achieve
>
>Automatic path migration is one advantage that can be derived through
>IPoIB-RC. 
>
>> this sort of connection transparent as to whether an IB or
>> IP network is in use then SDP and DAPL would seem to be more
>> appropriate.
>
>If the underlying connection failover is transparent, then the
>solution is transparent to the upper layer - whatever it may be
>IP/SDP/DAPL. How does it make one protocol more appropriate than the
>other? 
>
>> 
>> If the goal is to make a byte or message stream ULP
>> connection over an IB network, what benefit is there in
>> insisting on tunneling that connection in IP format?
>
>We are talking about IP over different IB modes and the benefits that
>accrue to the IP application such as APM and large MTU. You seem to be
>considering replacing IP and using a different ULP. Those are
>different discussions.
>
>> 
>> If IPoIB is going to expand its charter, it should be on the
>> basis of solving ULP problems, not merely trying to make it
>> a universal solution. That becomes an effort in "fixing" IB
>> to be more IP like. In my opinion, the end result will be
>> unappealing to any target market.
>
>I for one am interested in a failover of the type mentioned. It really
>doesn't matter at which level it is done i.e. at IB level or in IPoIB.
>If it can be done at IB level then it doesn't belong in IPoIB but if
>it cannot be, then we could consider it here. Does anyone have any
>solutions in either space?
>
>> 
>> One legitimate goal that has been cited was supporting a
>> larger MTU to allow atomic delivery of larger IP frames, at
>> least matching jumbo Ethernet frames. I think that RD and UC
>> should be considered for this goal. They meet the objective
>> while imposing fewer additional semantics than found in the
>> orignal link layer.
>
>Yes, one gets larger MTU and in the case of RC/RD Automatic path
>migration. Any connection/QPN mapping to a particular service reduces
>the lookup/scheduling needed in a server.
>
>I have no objection to considering RD and UC but I don't think they
>are in any way 'semantically closer' in context of IPoIB. They are all
>'connection' oriented links setup using CM REQ/REP as is the case with
>RC. They are point to point exactly as is IPoIB-RC and don't support
>multicast. As far as IPoIB is concerned all these modes are
>equivalent. If we describe the method for IPoIB over one of these
>modes the method can be trivially extended to the other modes. See
>section 3.0 in draft-kashyap-ipoib-advanced-capabilities-00.txt.
>
>Vivek
>
>> 
>> 
>> 
>> 
>> 
>> Caitlin Bestler
>> http://www.asomi.com/CaitlinBestler/
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Mon, 14 Oct 2002 14:03:34 -0400

Hi Vivek,

It looks like RC support in the CM is optional:

1.0.a vol 1 p.821
Table 254 Communication Management ClassPortInfo:CapabilityMask
9 IsReliableConnectionCapable Reliable Connections are supported
10 IsReliableDatagramCapable Reliable Datagrams are supported

While this changes some of the details of what you wrote, I don't
think it materially affects your conclusions.

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, October 14, 2002 1:01 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Ipoverib] recharter consideration

> what benefit it provides?

I agree, it will be useful to get an outline of a possible solution.

> Vivek) thinks this is useful enough to be considered?

A short comparison:

    QP based demuxing is the smallest delta over IPoIB-UD, followed by
    UC that adds RDMA Write to QP demultiplexing but drops multicast
    support. UC's large MTU support is deficient since it is an
    unreliable mode and hence not of any help in case of lost
    fragments. RC and RD go beyond UC by providing Large MTU, RDMA
    Read/Write, Automatic Path Migration, reliability. RD has the
    restriction that no new message can be transmitted from a given QP
    unless the previous message has been acknowledged by the peer.
    Additionally all HCAs are required to support RC but not RD.

In my opinion IPoIB-RC is clearly the most useful mode. UC doesn't
seem useful, QP demux would be a niche use, RD I can't say but am not
too much inclined towards since it is not supported by all HCAs. I
guess we could consider modes other than RC as long as there is some
usefulness since similar steps will be required to setup any of the
above modes. All require the use of CM to setup a 'connection'. But
considering all the issues RC is the most promising.

Vivek

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Mon, 14 Oct 2002 14:54:51 -0400

Actually, in terms of the HCA transport requirements, what I see is the
following:

1.0.a vol 1 p.831
Table 256 Channel Adapter Attributes
Transport Services Supported
HCA
HCAs shall be capable of supporting
the Unreliable Datagram, Reliable
Connection, and Unreliable
Connection transport service on
any QP supported by the HCA.

p.833
C17-11: HCAs shall be capable of supporting the Unreliable Datagram,
Reliable Connection, and Unreliable Connection transport service on any
QP supported by the HCA.

What HCA vendors are doing in this area may be another story.

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, October 14, 2002 2:53 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Yes, that is true. CM is needed for RC or any of the non-UD modes. 

I only mean that one can add the CM software module if the one
available doesn't support RC mode since the hardware will always
support it. But since RD is not required to be supported just getting
the right software is not enough - the right kind of HCA must also be
acquired. 

Vivek 

> 
> Hi again Vivek,
> 
> Yes, but if CM doesn't support the transport, I am not sure whether it
> matters if the HCA supports the transport as it is not usable (without the
> CM support). Wouldn't IPoIB in non UD modes rely on CM (and hence rely on
> the CM's ClassPortInfo) ?
> 
> -- Hal
> 
> Sent: Monday, October 14, 2002 2:32 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> Hal,
>     I think this is a software module only indicator. So these flags
>     indicate the capabilities of the supported CM(a software module)
>     but the specification otherwise requires all HCAs to support the
>     Reliable connected mode.
> 
> Vivek
> 
> > 
> > Hi Vivek,
> >

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Mon, 14 Oct 2002 14:54:51 -0400

1.0.a vol 1 p.831

p.833

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, October 14, 2002 2:53 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Vivek

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Mon, 14 Oct 2002 14:28:40 -0400

Hi again Vivek,

-- Hal

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, October 14, 2002 2:32 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Hal,

Vivek

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 14 Oct 2002 13:33:22 -0700 (PDT)

>considering all the issues RC is the most promising.

Putting unreliable datagrams through a reliable link layer may not be
a good fit in all cases. E.g. latency sensitive UDP traffic may only
need a best effort delivery and may not want the link-layer in-order
delivery algorithm to get in the way. UDP/IP/UC will enable the use
of large messages to avoid IP fragmentation.

So I think we should consider both UC and RC.

Jerry

---

From: "Vivek Kashyap" <[email-masked]>
Date: Wed, 16 Oct 2002 15:50:59 -0700

Mike,

We can consider both RC and UC. Other than their intrinsic link layer
qualities or modes both RC and UC look the same to the IP layer - similar
setup and use.

[ UC will not do well in face of a loss since the receiver just drops the
fragment and the rest of the message silently. It might be better to use
smaller MTU based UD based transfer since a loss of small data  can be
overcome by the application.  It is also better to use smaller packets  if
low latency is desired. Also if suitable buffering is done by the
application RC can easily retransmit the missing fragment in 1 RTT
providing a better service than UC which will drop all data. The data we
are talking about is based on large MTUs - 32 K of IP MTU implies 16
fragments in a single UC link for a 2K link. With all this I still don't
have an argument with using UC for streaming data if found appropriate in
certain situations but UC is not suitable for all applications. ]

Vivek

--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Michael Krause                                                                                                   
                      <[email-masked]        To:       "H.K. Jerry Chu" <[email-masked]>, [email-masked]               
                      m>                       cc:       [email-masked]                                                             
                      Sent by:                 Subject:  Re: [Ipoverib] recharter consideration                                        
                      ipoverib-admin@ie                                                                                                
                      tf.org                                                                                                           

                      10/14/2002 03:08                                                                                                 
                      PM                                                                                                               

>So I think we should consider both UC and RC.

I've never seen a reason to support RC for IP-based traffic as the upper
layers already assume a lossy fabric.  If we were to support any
improvements, I would hazard that UC provides all of the requisite
functionality without any additional baggage or transport functionality
replication.  Keep in mind that IB is essentially loss-less for connected
traffic sans CRC errors thus why be burdened with RC.

BTW, the retransmission timers are allowed to be fairly large in IB fabrics

making them unsuitable for real-time traffic such as streaming media unless

you set them to a small value which may trigger false retransmission
events.  I certainly don't want to see unreliable traffic flowing over a RC

link for such workloads.

Mike

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Thu, 17 Oct 2002 11:26:42 -0400

Hi,

I would like to elaborate on other causes of dropping (loss) in an IB subnet
in the face of congestion. Aside from the CRC errors, when either Head of
Queue lifetime or switch lifetime are exceeded in IB switches these packets
are also dropped and need to be included in the discussion of loss. This
brings in the notion of traffic patterns.

Wouldn't RC have benefit in such highly utilized fabrics and UC might not as
entire messages would need to be retransmitted rather than an individual
segment of a message ?

-- Hal

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Thursday, October 17, 2002 9:57 AM
To: Vivek Kashyap
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

>certain situations but UC is not suitable for all applications. ]

Could not more strongly disagree - UC is all that is required:

- IB is not the Internet, i.e. it is a controlled / well-understood fabric 
within a data center

- IB is a strongly ordered, essentially loss-less fabric

- IB only drops packets due to a CRC error

- CRC rates should be in the 10^-12 to 10^-15 range with most engineers 
believing 10^-15 will be common for deployments.

- Extremely large percentage of packets will be less than 2KB for a variety 
of workloads.

- Use of UC eliminates ACK packets being exchanged thus reducing fabric 
workload and increasing overall fabric efficiency.

- IB components have thin buffering and depending upon the workload and 
number of QP operating over a given CA, they may also have thin memory 
resources to accommodate retransmission.

- It is simpler and easier for a TCA to implement UC rather than RC as it 
does not have to maintain much in terms of state or resources and can more 
easily flow through to its secondary fabric as a result.

Conclusion: UC will work just fine as RC will provide no value for IP 
workloads and in fact, will result in lost fabric efficiency and 
potentially increased probability of congestion events.

Mike

---

From: "Rosenstock, Hal" <[email-masked]>
Date: Thu, 17 Oct 2002 11:26:42 -0400

Hi,

-- Hal

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Thursday, October 17, 2002 9:57 AM
To: Vivek Kashyap
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Ipoverib] recharter consideration

Mike

---

From: Michael Krause <[email-masked]>
Date: Thu, 17 Oct 2002 09:29:24 -0700

>entire messages would need to be retransmitted rather than an individual

The drop times for HOL blocking in an individual switch element can be 
quite large, i.e. measured in seconds if desired.  In general, this type of 
drop will not occur if all components are acting well, i.e. leaf 
destination endnodes are consuming packets and returning flow control.

I will also note that if the fabric is this congested, then adding in the 
additional packets for RC acknowledgement only makes things worse not 
better.  As the IBTA works out congestion control and QoS, it is quite 
likely that this type of drop will become even rarer as the injection rates 
and event notification will present to prevent such occurrences.

Mike