Subject: RE: [Ipoverib] Yokohama IETF

From: Yaron Haviv <[email-masked]>
Date: Sun, 7 Jul 2002 21:25:23 +0300

Mike, Jerry

I understand that there are situations where checksum offloading is
difficult to achieve
in the generic sense and has potential problems
However there is more than one case where it could be used and will bring
much value
e.g. in case of gateways, routers, xCA's supporting HW offload
Since in RC the connection parameters can be negotiated, we can have offload
when, and in the way that it could be verified 
we shouldn't block the cases where it could be easily supported from doing
so 
I think relaying on the HCA to do the work for us is a mistake, it wont
happen in the near future, it also requires Verbs changes to tell the HW
which QP should support offload, and an interface should be determined to
tell the apps of errors, etc'

Just as an example an Ethernet TCA can verify the checksum for Ethernet
packets if ok pass, if not discard, in any case leave the original checksum
values in case the traffic will find it self routed to somewhere not aware
of the offload
on the other side when it negotiated that it supports checksum offload it
can get the packet without the checksum and update the correct checksum in
the outgoing Ethernet packets 
This scenario is not limited to TCA's only, it could be used in any case
where routed packets get the right treatment, or not routed at all, and this
could be negotiated during the RC establishment, where by default the
Checksum is always generated   

What holes do you see in that ?? 
As you know most OS's support that feature today, so it doesn't involve
changes to the TCP/IP stack
Checksum offloading is critical especially in kernel apps that don't do an
additional copy, we don't want to limit InfiniBand and make it perform less
than Ethernet with a simple NIC (which has offload features), so where we
can lets do use it 

Yaron

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Wednesday, July 03, 2002 7:26 AM
To: Jerry Chu [expiry 14 July 02]; [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] Yokohama IETF

At 01:12 PM 7/3/2002 +0800, Jerry Chu [expiry 14 July 02] wrote:
>Do we believe this is architecturally feasible at all, given the
>end2end nature of TCP/UDP checksum, and the presence of middlebox
>or tunneling that makes the determination of where TCP/UDP pkts
>actually terminate tricky? (One must ensure the complete path lies
>within a single IB RC.)

Been tried many times and no one has come up with a good solution.  I don't
think this is a worthwhile effort to pursue.  Put a checksum off-load
engine within the CA and do some fix-up if need be just like what is done
for Ethernet and we don't need to specify anything here to solve this
problem.

Mike

>Jerry
>
> > Vivek,
> >
> > Maybe we can add to the IPoIB RC a feature we once discussed
> > For TCP/UDP checksum offloading no need to specify the exact method now
> > Just mention it is a possible future goal
> >
> > Yaron
> >
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On Behalf
Of
> > Vivek Kashyap
> > Sent: Friday, June 28, 2002 1:45 AM
> > To: [email-masked]
> > Cc: 'Jerry Chu'; 'ipoverib'
> > Subject: Re: [Ipoverib] Yokohama IETF
> >
> > Jerry, Yaron,
> >
> >         As you know these ideas have been proposed by me in the draft:
> >
> >         draft-kashyap-ipoib-advanced-capabilities-00.txt.
> >
> >         I had presented a set of slides at IETF-53. See slides 16 to 24
at:
> >
> >         http://www.ietf.org/proceedings/02mar/slides/ipoib-1/index.html
> >
> > Vivek
> >
> > PS. I won't be attending IETF-54.
> >
> > >
> > > Jerry,
> > >
> > > Regarding the extra topics:
> > > I think the IPoIB over RC is important to improve both performance and
> > > robustness
> > >
> > > The demuxing to few QP's might be useful too for some applications
> > >
> > > Another thing that IB ARP can help in is detecting SDP (Socket Direct)
> > > clients e.g. through a capability bit
> > > I'm not sure if it requires changes in the WG charter
> > >
> > > Yaron
> > >
> > > -----Original Message-----
> > > From: [email-masked] [mailto:[email-masked]]On
> Behalf Of
> > > Jerry Chu
> > > Sent: Friday, June 28, 2002 12:57 AM
> > > To: ipoverib
> > > Subject: [Ipoverib] Yokohama IETF
> > >
> > > With the Yokohama meeting fast approaching, it is
> > > time to prepare for the agenda. We need to first
> > > decide if there are enough topics to discuss. What I
> > > need from you are -
> > >
> > > 1. how many of you are planning to attend the meeting
> > > 2. how many have slides to present (including the topic)
> > >
> > > Please let me know asap. Even if you can't make it to
> > > Yokohama in person, you can still make slides and give
> > > them to me. I will present the slides on your behalf.
> > >
> > > On a separate topic, the three basic drafts for IPoIB communication
> > > listed below
> > >
> > > draft-ietf-ipoib-link-multicast-02.txt
> > > draft-ietf-ipoib-ip-over-infiniband-01.txt
> > > draft-ietf-ipoib-dhcp-over-infiniband-01.txt
> > >
> > > have gone through several rounds of review and seem
> > > to be in pretty good shape for a WG last call, pending
> > > the resolution of the promisucous multicast mode
> > > support. Bill and I would like to see if there is
> > > enough interest to extend the current WG chart to
> > > cover a number of more advanced features such as
> > >
> > > 1. running IP over non-UD IB transports
> > > 2. demultiplexing IP packets into multiple QPs based
> > > on, e.g. layer 4 fields
> > >
> > > This is the time to bring up any remaining topics
> > > you may have for the WG to cover. Note that in order
> > > to get the permission from the ADs to add any work
> > > group item to the current charter, the item must be well-defined and
> > > scoped, with enough interest from the WG to support the work.
> > >
> > > Your friendly co-chairs
> > > Jerry and Bill
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > >
> > >
> > > _______________________________________________
> > > IPoverIB mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/ipoverib
> > >
> >
> >
> > --
> > Vivek Kashyap
> > Linux Technology Center, IBM
> > [email-masked]
> > [email-masked]
> > 503 578 3422 (o)
> >
> > _______________________________________________
> > IPoverIB mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ipoverib
> >
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Mon, 08 Jul 2002 06:30:00 -0700

At 09:25 PM 7/7/2002 +0300, Yaron Haviv wrote:
>difficult to achieve in the generic sense and has potential problems 
>much value e.g. in case of gateways, routers, xCA's supporting HW offload 
>Since in RC the connection parameters can be negotiated, we can have 
>offload when, and in the way that it could be verified we shouldn't block 
>the cases where it could be easily supported from doing so  I think 
>relaying on the HCA to do the work for us is a mistake, it wont happen in 
>the near future, it also requires Verbs changes to tell the HW which QP 
>should support offload, and an interface should be determined to tell the 
>apps of errors, etc'

It is not possible to achieve in a generic sense else we would have done it 
years ago when cluster interconnects were first developed.  Nothing to date 
has changed that capability.

If a TCA acting as an IP terminus point implements checksum off-load this 
is great but that has no impact on IP over IB which does not deal with such 
a situation.

It does not require Verbs changes unless you want it standard from the IBTA 
and that is most likely not worthwhile to pursue at this time.

>packets if ok pass, if not discard, in any case leave the original 
>checksum values in case the traffic will find it self routed to somewhere 
>not aware of the offload on the other side when it negotiated that it 
>supports checksum offload it can get the packet without the checksum and 
>update the correct checksum in the outgoing Ethernet packets  This 
>scenario is not limited to TCA's only, it could be used in any case where 
>routed packets get the right treatment, or not routed at all, and this

If people want to define an Ethernet TCA wire protocol that provides such 
services, see RNDIS or create your own as RNDIS can only be used on one 
OS.  However, the creation of such a wire protocol is outside of the scope 
of IP over IB, IMO.

>changes to the TCP/IP stack Checksum offloading is critical especially in 
>kernel apps that don't do an additional copy, we don't want to limit 
>InfiniBand and make it perform less than Ethernet with a simple NIC (which 
>has offload features), so where we can lets do use it

You have not solved really anything with the above that has not been 
considered before and determined to be a special before.

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Mon, 08 Jul 2002 18:44:44 +0200

Mike,
You have referred only to one scenario: TCA
IPoIB as far as I know is the only standard for routing IP traffic in IB
As such it is totally legitimate to build a router that would route IP
traffic (e.g. from Ethernet) to IB based on IPoIB
RNDIS has certain usage model, it is a L2 solution, it works as an I/O
channel owned by its host, unlike routers
Some people may choose to have an RNDIS or a like IOC some might choose
otherwise for various reasons

My proposal helps to increase performance, without going through too much
hassle
It is not intended to be generic as I mentioned, just negotiated at
connection initiation phase
You may choose not to use it; it works in a way that it breaks nothing

This whole email was to response on Jerry's WG scope email
I don't see why we can't add this offload as optional part of the standard,
just like RC & QPN, which has much less impact on performance

you mention I haven't solved anything, what is there to solve with such an
approach ?
I don't see the reason for opposing to such a capability that has such
impact on performance
And could be implemented quite easily (much easier than doing it at the HCA)

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Michael Krause
Sent: Monday, July 08, 2002 3:30 PM
To: Yaron Haviv
Cc: [email-masked]'
Subject: RE: [Ipoverib] Yokohama IETF

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 08 Jul 2002 09:31:06 -0700

At 06:44 PM 7/8/2002 +0200, Yaron Haviv wrote:
>hassle It is not intended to be generic as I mentioned, just negotiated at
>connection initiation phase You may choose not to use it; it works in a 
>way that it breaks nothing

At connection initiation one does not know whether the packet is to be 
routed via a different I/O through the endnode so such negotiation does not 
solve this problem.

>And could be implemented quite easily (much easier than doing it at the HCA)

Within a HCA it is just a 1's complement with minimal software fix-up 
within the host just like it is for Ethernet today.  The data is 
communicated as part of the completion event so there is nothing expensive 
to do here - it is quite trivial and I expect to see it added to most HCA 
as IB ramps to volume.  Thus for any QP type being used for IP 
communication, one can have this available without requiring anything from 
a standards group such as this.

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Mon, 08 Jul 2002 19:55:33 +0200

Mike,

If you look at my proposal in the original email you will see that:
1. The router in this case will verify the checksum for incoming (e.g. from
Ethernet) and will keep the original value, in this case if it is routed
through other I/O's there is no problem, since it is a completely valid
TCP/IP packet with the right checksum value in place
2. For outgoing packets, during the connection setup the router will tell
the node it will do the checksum for it (only if he is sure he can cover all
the routing cases) than the node will send the packets to the router with
only the pseudo header checksum, the router will fill the checksum

So in that case it does solve the problem since there won't be any illegal
TCP/IP packets going anywhere other than it is supposed to
We are focusing on the router example, however it could hold for other cases
too

Yaron

-----Original Message-----
From: Michael Krause [mailto:[email-masked]]
Sent: Monday, July 08, 2002 6:31 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [Ipoverib] Yokohama IETF

Mike

---

From: Yaron Haviv <[email-masked]>
Date: Mon, 08 Jul 2002 20:15:48 +0200

Mike,

Another thing, indeed the Rx side might be simple
However in the transmit side you have to buffer the entire TCP frame so you
could place the checksum at the front
In case of IPoIB RC the massage could be long (up to 64K), and also consist
of several IB frames
Buffering a complete message at the HCA contradicts the IB HCA model
I'm not sure that the current HCA architectures can just do that

If you want to limit the IPoIB RC to just one 2K frame, I think you may miss
its target

Yaron

-----Original Message-----

Mike

---

From: Michael Krause <[email-masked]>
Date: Mon, 08 Jul 2002 21:32:11 -0700

>Another thing, indeed the Rx side might be simple

It is quite simple and works well for gateway / router solutions.

>I'm not sure that the current HCA architectures can just do that

I was not aware that checksum off-load for RC was the only focus of the 
discussion.  Clearly a UD solution can provide checksum off-load fairly 
easily within a HCA as well similar to what is done for Ethernet so that 
does not seem to be an issue.

>If you want to limit the IPoIB RC to just one 2K frame, I think you may miss

An examination of workloads will show the 80/20 (small / large) rule for 
message size for inbound and 80/20 (large / small) for outbound.  Large in 
most cases is under 2KB for 90% of the data moved.  Large TCP send was used 
to fake out the sending side so that it would allow the hardware to perform 
the segmentation and checksum calculations using small data movements while 
the host stack implemented in software saw this as a large MSS.  One can 
use a similar solution for IB if this is really required.

Another approach is to use the trailer checksum approach that HP proposed 
to the IETF about 10-15 years ago that allowed one to transmit the checksum 
as a trailer to the packet.  Software could fix up the receiving side quite 
nicely before passing it up to the upper layers thus allaying any concerns 
without complicating the HCA logic by that much - not really any different 
than what is used for receive in such a case. This would work for RC with 
any size packet.

Mike