Subject: [Ipoverib] Checksum offload for IPoIB RC

From: Yaron Haviv <[email-masked]>
Date: Mon, 15 Jul 2002 23:42:26 +0200

I have followed the emails in the reflector, it seems like some people
haven't read through my emails, before responding , I would appreciate it if
you can read this one carefully and comment on it
Some of the problems that some of you raised are addressed, and some doesn't
apply to what I suggest but to older/other suggestions people had

One of the main loads in TCP/IP processing is calculating the checksum since
it requires going through each one of the data bytes in the payload reading
it from SDRAM memory and calculating the checksum
Another side effect is that the CPU cache is filled with unimportant data
which has its effect on system performance too
Most stacks today need to do a copy between the TCP/IP buffers and the
application memory, during this process they usually calculate the checksum
to save overhead
In applications/stacks that can directly read the stack buffers (e.g. kernel
apps, appliances, etc') the L4 checksum calculation could be over 40% of the
processing overhead
(Source: TCP/IP at Near-Gigabit Speeds, Andrew G, Jeff C, and Ken Y, Duke
University, Dec 1999)
This is why most GbE NIC's implement checksum calculation in HW, and most
TCP/IP stacks already support that

In IB we have 2 problems when wanting to implement it in the CA HW
1.	the IPoIB protocol is only one of many protocols that the HCA needs to
support, it is using a special QP (or more) that are determined on the fly,
and a lot of flexibility exists and will be added in the future
2.	in RC we can send/receive messages which consist of several IB Frames
(there are many DC applications that can make good use of packets larger
than 1K, e.g. NFS/CIFS), in TCP/UDP the checksum is in the beginning, so we
cant send the packets before we buffered the entire message in the HCA, this
contradicts the basic HCA model, as well as difficult to achieve (since
HCA's are not built in a store & Fwd model like Ethernet)
(as to Mike suggestion to use a specific RFC for that, as far as I
understand no one uses it in Ethernet, so why should someone agree using it
in IB)

I assume that some HCA's will implement some of it in the future as IB and
IPoIB will mature and after they overcome their current problems, however
this will take significant time, probably start for UD traffic
We cannot let IB under perform GbE in such a significant rate during that
time

I believe that there are many cases where checksum offload/bypass can be
used
And without creating holes and potential problems

The method I proposed was to avoid doing checksum ONLY in IPoIB RC
And ONLY after it was decided how/if between both peers during the RC
establishment

The main problem we need to overcome is to make sure that no illegal TCP/IP
packets (without L4 checksum) travel  outside an established IPoIB RC TUNNEL
By either having a valid checksum in the packet, or by agreeing there is no
valid L4 checksum and the receiving side will fill it in case it forwards
the packet to another machine/stack

a simple scenario where we can use it very easily is in the case of an IPoIB
Router/TCA that forwards traffic from Ethernet to IB
when it receives the Ethernet traffic it verifies the checksum, still
keeping the valid checksum in the packet and forwards it to the IB server
even if the IB server will accidentally forward the packet to somewhere
else, no harm was done since it is a completely valid TCP/IP packet (with
checksum)
in the IB to Ethernet side, it signals to the host that it will do the
checksum for it (during the RC establishment), the host sends the packets
only with the pseudo header checksum, the router fills the checksum, and
only then forwards the packet
again the forwarded packet is a fully valid TCP/IP packet

other cases where it will work:
1. nodes that don't forward traffic, or can guaranty that they will complete
the checksum in case they do
2. hosts that will use a CA that offloads checksum (only on one side)

the node can choose by default not to take part of this option like they can
with all the rest of the options (QPN, etc..)
the nodes will need to answer 2 questions during the RC establishment
process
1.	can I send you packets, after verifying the L4 checksum in HW (keeping a
valid packet)
2.	can I accept packets with no checksum and calculate it myself, and also
guarantee that I will forward it with the checksum in place in all cases
this will help determining if/how to use this method

Thanks, Yaron

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Bill Strahm" <[email-masked]>
Date: Mon, 15 Jul 2002 15:10:32 -0700

I have been considering this issue for a LONG time now, and I think I
might be ready to open my mouth and utter something intelligent.

I am going to spout off Bill Strahm's Personal Opinion here, so this is
not from a WG chair.

I see little need to standardize Checksum offload capabilities between
NICs/HCAs/TCAs and host IP stacks.  This has all ready been done by most
hardware/OS vendors that I am aware of.  Also because the connection
between a Host IP stack and a NIC/HCA will not be a wire protocol (and
don't try to include bus wires here, I'm not buying that one) there is
not much that the IETF can do.  You might be able to claim something
else with the TCA case, but I tend to view IB as an extension of the
host bus in this case and still not the perview of the IETF.

The second thing that is discussed below is to special case TCP Checksum
offloading and not including it in the packet.  I am STRONGLY opposed to
this on several grounds, the first is that it isn't expensive to put it
into hardware (20 dollar commodity nics are including this now, so it
can't be all that hard), and the second because there has been a long
history of though that has gone into TCP processing, and I would hate to
special case stacks for the IB case (I think this is Eric N's argument).

You talk about large IP packets under RC, however I think you will find
the optimal solution is still close to your basic IB MTU, so making the
IP packet larger than the IB MTU is only going to decrease throughput.
I could be wrong, but until I see real packet traces with my own eyes,
I'm not sure I could buy it.  So I am back down to a TCP MSS that is
either 2K or 4K, this doesn't seem to be excessive for checksum
processing.   Your TCP segment will definitely go down to this range if
you were to leave IB and go onto a more commong 1.5K MTU.

WG Chair hat back on

1) What does the WG think, I have seen notes from our AD that this
probably isn't a worth while optimization, and they will likely be
resistance at the IESG level for proposing a TCP-Nochecksum option.

2) Can anyone get me a real transistor count (to the nearest K maybe) to
implement a generic TCP Checksum offload function ?  I know that many
vendors have this in their current 100M, 1G ethernet NICs.  Are we
talking 100 transistor, 1000, 10000, 100000.  My guess is that it is
order of magnitude 10K, but I have nothing to back that up.

3) How is the Sushi in Japan ?  Glad I'm not eating raw fish <G>

Billb

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Yaron Haviv
Sent: Monday, July 15, 2002 2:42 PM
To: [email-masked]
Cc: 'Vivek Kashyap'; [email-masked]; [email-masked]
Subject: [Ipoverib] Checksum offload for IPoIB RC

I have followed the emails in the reflector, it seems like some people
haven't read through my emails, before responding , I would appreciate
it if you can read this one carefully and comment on it Some of the
problems that some of you raised are addressed, and some doesn't apply
to what I suggest but to older/other suggestions people had

One of the main loads in TCP/IP processing is calculating the checksum
since it requires going through each one of the data bytes in the
payload reading it from SDRAM memory and calculating the checksum
Another side effect is that the CPU cache is filled with unimportant
data which has its effect on system performance too Most stacks today
need to do a copy between the TCP/IP buffers and the application memory,
during this process they usually calculate the checksum to save overhead
In applications/stacks that can directly read the stack buffers (e.g.
kernel apps, appliances, etc') the L4 checksum calculation could be over
40% of the processing overhead
(Source: TCP/IP at Near-Gigabit Speeds, Andrew G, Jeff C, and Ken Y,
Duke University, Dec 1999) This is why most GbE NIC's implement checksum
calculation in HW, and most TCP/IP stacks already support that

In IB we have 2 problems when wanting to implement it in the CA HW
1.	the IPoIB protocol is only one of many protocols that the HCA
needs to
support, it is using a special QP (or more) that are determined on the
fly, and a lot of flexibility exists and will be added in the future
2.	in RC we can send/receive messages which consist of several IB
Frames
(there are many DC applications that can make good use of packets larger
than 1K, e.g. NFS/CIFS), in TCP/UDP the checksum is in the beginning, so
we cant send the packets before we buffered the entire message in the
HCA, this contradicts the basic HCA model, as well as difficult to
achieve (since HCA's are not built in a store & Fwd model like Ethernet)
(as to Mike suggestion to use a specific RFC for that, as far as I
understand no one uses it in Ethernet, so why should someone agree using
it in IB)

I assume that some HCA's will implement some of it in the future as IB
and IPoIB will mature and after they overcome their current problems,
however this will take significant time, probably start for UD traffic
We cannot let IB under perform GbE in such a significant rate during
that time

I believe that there are many cases where checksum offload/bypass can be
used And without creating holes and potential problems

The method I proposed was to avoid doing checksum ONLY in IPoIB RC And
ONLY after it was decided how/if between both peers during the RC
establishment

The main problem we need to overcome is to make sure that no illegal
TCP/IP packets (without L4 checksum) travel  outside an established
IPoIB RC TUNNEL By either having a valid checksum in the packet, or by
agreeing there is no valid L4 checksum and the receiving side will fill
it in case it forwards the packet to another machine/stack

a simple scenario where we can use it very easily is in the case of an
IPoIB Router/TCA that forwards traffic from Ethernet to IB when it
receives the Ethernet traffic it verifies the checksum, still keeping
the valid checksum in the packet and forwards it to the IB server even
if the IB server will accidentally forward the packet to somewhere else,
no harm was done since it is a completely valid TCP/IP packet (with
checksum)
in the IB to Ethernet side, it signals to the host that it will do the
checksum for it (during the RC establishment), the host sends the
packets only with the pseudo header checksum, the router fills the
checksum, and only then forwards the packet again the forwarded packet
is a fully valid TCP/IP packet

other cases where it will work:
1. nodes that don't forward traffic, or can guaranty that they will
complete the checksum in case they do 2. hosts that will use a CA that
offloads checksum (only on one side)

the node can choose by default not to take part of this option like they
can with all the rest of the options (QPN, etc..) the nodes will need to
answer 2 questions during the RC establishment process
1.	can I send you packets, after verifying the L4 checksum in HW
(keeping a
valid packet)
2.	can I accept packets with no checksum and calculate it myself,
and also
guarantee that I will forward it with the checksum in place in all cases
this will help determining if/how to use this method

Thanks, Yaron

_______________________________________________
IPoverIB mailing list
[email-masked] https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 16 Jul 2002 01:56:51 +0200 (CEST)

> apply to what I suggest but to older/other suggestions people had

Sorry for not reading all the emails.
I looked at the charter for ipoib and it doesn't seem to say that this WG
is chartered to change the part of the Internet architecture that says
we have end-to-end tranport checksums.
Was I reading the wrong charter? (tongue firmly planted in cheek)

BTW: The issue about skipping the TCP checksum has been raised by when various
new technologies have been specified to carry IP. I think (but I'm not certain)
that this was most recently raised in the IP over ATM context. History shows
that the IETF has provided a consistent answer to this question.

> We cannot let IB under perform GbE in such a significant rate during that

This seems to be an implementation issue. What does it have to do with
the Internet architecture?

  Erik