Subject: RE: [Ipoverib] Checksum offload for IPoIB RC

From: Yaron Haviv <[email-masked]>
Date: Tue, 16 Jul 2002 21:07:48 +0300

Bill,

We are facing 3 separate issues 
1. Is checksum offloading needed? Clearly the answer is yes (if you want I
will direct you to more than one link on this subject)
2. What should be the solution ? in HCA ? Something else ?
3. Is my solution feasible and complete 

I haven't seen people saying that my solution is not going to work (in the
scenarios I described), if someone does so please respond 
My solution doesn't require ANY changes to the TCP Stack as opposed to what
some suggested 

Regarding the HCA Vs. my proposal
First they don't contradict each other but complement 
I believe that in RC you cannot limit the user to use a single framed packet
Some people will try, and even might benefit from using larger IP packets
4/8K over a 2K MTU 
The problem is not the 10K gates, but a different architecture needed, if
the HCA vendors can refer to that on the reflector

Since as you noted IB could also be viewed as an internal bus we are
somewhat different than ATM, GbE, ...
This is why we have for example a connection establishment process in
Vivek's draft
Or features like QPN that do not exist in Ethernet, or various multicasting
requirements  
Maybe this offload capability could just be viewed as another capability of
the fabric as many other things we went through are
Or as an IB extension to IPoIB for attaching offload capable devices  

Yaron
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Bill Strahm
Sent: Tuesday, July 16, 2002 12:11 AM
To: Yaron Haviv
Cc: [email-masked]
Subject: RE: [Ipoverib] Checksum offload for IPoIB RC

I have been considering this issue for a LONG time now, and I think I
might be ready to open my mouth and utter something intelligent.

I am going to spout off Bill Strahm's Personal Opinion here, so this is
not from a WG chair.

I see little need to standardize Checksum offload capabilities between
NICs/HCAs/TCAs and host IP stacks.  This has all ready been done by most
hardware/OS vendors that I am aware of.  Also because the connection
between a Host IP stack and a NIC/HCA will not be a wire protocol (and
don't try to include bus wires here, I'm not buying that one) there is
not much that the IETF can do.  You might be able to claim something
else with the TCA case, but I tend to view IB as an extension of the
host bus in this case and still not the perview of the IETF.

The second thing that is discussed below is to special case TCP Checksum
offloading and not including it in the packet.  I am STRONGLY opposed to
this on several grounds, the first is that it isn't expensive to put it
into hardware (20 dollar commodity nics are including this now, so it
can't be all that hard), and the second because there has been a long
history of though that has gone into TCP processing, and I would hate to
special case stacks for the IB case (I think this is Eric N's argument).

You talk about large IP packets under RC, however I think you will find
the optimal solution is still close to your basic IB MTU, so making the
IP packet larger than the IB MTU is only going to decrease throughput.
I could be wrong, but until I see real packet traces with my own eyes,
I'm not sure I could buy it.  So I am back down to a TCP MSS that is
either 2K or 4K, this doesn't seem to be excessive for checksum
processing.   Your TCP segment will definitely go down to this range if
you were to leave IB and go onto a more commong 1.5K MTU.

WG Chair hat back on

1) What does the WG think, I have seen notes from our AD that this
probably isn't a worth while optimization, and they will likely be
resistance at the IESG level for proposing a TCP-Nochecksum option.

2) Can anyone get me a real transistor count (to the nearest K maybe) to
implement a generic TCP Checksum offload function ?  I know that many
vendors have this in their current 100M, 1G ethernet NICs.  Are we
talking 100 transistor, 1000, 10000, 100000.  My guess is that it is
order of magnitude 10K, but I have nothing to back that up.

3) How is the Sushi in Japan ?  Glad I'm not eating raw fish <G>

Billb

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Yaron Haviv
Sent: Monday, July 15, 2002 2:42 PM
To: [email-masked]
Cc: 'Vivek Kashyap'; [email-masked]; [email-masked]
Subject: [Ipoverib] Checksum offload for IPoIB RC

I have followed the emails in the reflector, it seems like some people
haven't read through my emails, before responding , I would appreciate
it if you can read this one carefully and comment on it Some of the
problems that some of you raised are addressed, and some doesn't apply
to what I suggest but to older/other suggestions people had

One of the main loads in TCP/IP processing is calculating the checksum
since it requires going through each one of the data bytes in the
payload reading it from SDRAM memory and calculating the checksum
Another side effect is that the CPU cache is filled with unimportant
data which has its effect on system performance too Most stacks today
need to do a copy between the TCP/IP buffers and the application memory,
during this process they usually calculate the checksum to save overhead
In applications/stacks that can directly read the stack buffers (e.g.
kernel apps, appliances, etc') the L4 checksum calculation could be over
40% of the processing overhead
(Source: TCP/IP at Near-Gigabit Speeds, Andrew G, Jeff C, and Ken Y,
Duke University, Dec 1999) This is why most GbE NIC's implement checksum
calculation in HW, and most TCP/IP stacks already support that

In IB we have 2 problems when wanting to implement it in the CA HW
1.      the IPoIB protocol is only one of many protocols that the HCA
needs to
support, it is using a special QP (or more) that are determined on the
fly, and a lot of flexibility exists and will be added in the future
2.      in RC we can send/receive messages which consist of several IB
Frames
(there are many DC applications that can make good use of packets larger
than 1K, e.g. NFS/CIFS), in TCP/UDP the checksum is in the beginning, so
we cant send the packets before we buffered the entire message in the
HCA, this contradicts the basic HCA model, as well as difficult to
achieve (since HCA's are not built in a store & Fwd model like Ethernet)
(as to Mike suggestion to use a specific RFC for that, as far as I
understand no one uses it in Ethernet, so why should someone agree using
it in IB)

I assume that some HCA's will implement some of it in the future as IB
and IPoIB will mature and after they overcome their current problems,
however this will take significant time, probably start for UD traffic
We cannot let IB under perform GbE in such a significant rate during
that time

I believe that there are many cases where checksum offload/bypass can be
used And without creating holes and potential problems

The method I proposed was to avoid doing checksum ONLY in IPoIB RC And
ONLY after it was decided how/if between both peers during the RC
establishment

The main problem we need to overcome is to make sure that no illegal
TCP/IP packets (without L4 checksum) travel  outside an established
IPoIB RC TUNNEL By either having a valid checksum in the packet, or by
agreeing there is no valid L4 checksum and the receiving side will fill
it in case it forwards the packet to another machine/stack

a simple scenario where we can use it very easily is in the case of an
IPoIB Router/TCA that forwards traffic from Ethernet to IB when it
receives the Ethernet traffic it verifies the checksum, still keeping
the valid checksum in the packet and forwards it to the IB server even
if the IB server will accidentally forward the packet to somewhere else,
no harm was done since it is a completely valid TCP/IP packet (with
checksum)
in the IB to Ethernet side, it signals to the host that it will do the
checksum for it (during the RC establishment), the host sends the
packets only with the pseudo header checksum, the router fills the
checksum, and only then forwards the packet again the forwarded packet
is a fully valid TCP/IP packet

other cases where it will work:
1. nodes that don't forward traffic, or can guaranty that they will
complete the checksum in case they do 2. hosts that will use a CA that
offloads checksum (only on one side)

the node can choose by default not to take part of this option like they
can with all the rest of the options (QPN, etc..) the nodes will need to
answer 2 questions during the RC establishment process
1.      can I send you packets, after verifying the L4 checksum in HW
(keeping a
valid packet)
2.      can I accept packets with no checksum and calculate it myself,
and also
guarantee that I will forward it with the checksum in place in all cases
this will help determining if/how to use this method

Thanks, Yaron

_______________________________________________
IPoverIB mailing list
[email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Michael Krause <[email-masked]>
Date: Sun, 21 Jul 2002 07:25:04 -0700

>will direct you to more than one link on this subject)

Measurements on GbE (we've done this within HP on multiple OS targets) 
typically show a 15% performance gain is achieved (when combined with 
complete copy avoidance, one can see performance gains of 30-40% depending 
upon implementation which is perhaps where you number came from).  This can 
be mitigated as you noted with a combined copy/checksum operation reducing 
the gain along the way.  Copies do occur for user-kernel crossings as well 
as kernel-kernel crossings since transfers are usually between different 
subsystems and the buffers were all "un-named / anonymous" from the target 
subsystem's perspective.

>2. What should be the solution ? in HCA ? Something else ?

Typical implementations for hardware are to perform a 1's complement of the 
entire byte stream and hand this value off as part of the completion event 
for a receive.  The driver then performs a quick fix-up to validate the 
packet and stores the result as required.  As such, a HCA implementation 
would be as simple as 1's complement a receive WQE as it is being DMA'ed 
from host-memory and if a receive, post that as part of the completion 
event or if a send, insert it at a specified off-set within the byte stream 
before sending it to the remote.  I agree that for outbound this can be 
complicated if the message is greater than 1 PMTU on the fabric thus there 
is some value in trying to not do this in hardware.  For some OS, the OS 
has come to the rescue  to mitigate the problem for data that is sourced 
from a buffer cache; for others there is no way to mitigate this problem.

Note: This all further complicated by IPv6 which cannot take as simple an 
approach as IPv4 since the next header allows various combinations of 
headers which hardware simply cannot deal with at all.  All vendors that 
I'm aware of will completely punt on checksum off-load if the next header 
is not the transport header.  I don't see this as changing any time soon 
(exception is when the solution is providing additional value-add 
functionality such as IPSec and thus the hardware already had to take quite 
a bit of information into account and the cost / benefit is clear).

>3. Is my solution feasible and complete

It is not clear it is worthwhile either.  IP over IB was intended to be a 
way to have a consolidated fabric within the set of endnodes thus 
eliminating the need for something like redundant Ethernets, etc.  The 
majority of the traffic was to flow over IB transports and not IP-based 
transports thus performance optimizations for IP-based were not the primary 
focus.  It seems to me that the workgroup would be best served by getting 
the specs through final call and implemented so that experience can be 
gained.  Once this is up and we have interoperability established, the 
issue can be raised once again.  In the mean time, one can take the issue 
up within the tvswg for some parallel discussion to gain guidance.

>I believe that in RC you cannot limit the user to use a single framed packet

Sure you can.  It is a ULP issue on top of RC and a ULP can limit a message 
to any size including 1 byte if it likes.

>Some people will try, and even might benefit from using larger IP packets

There is some support for large sends over TCP today but this does not 
change the fundamentals for what is on the wire and can be accomplished in 
a variety of implementation options.

>the HCA vendors can refer to that on the reflector

It is the gates; it will always be the gates.  Cost is the dominating 
factor in making any technical decision and one must understand the cost / 
benefit of a feature before investing time / resources in going through the 
next step.

Mike