Subject: RE: [Ipoverib] Yokohama IETF

From: "Jerry Chu [expiry 14 July 02]" <[email-masked]>
Date: Fri, 12 Jul 2002 15:14:47 +0800 (CST)

Yaron,

I'm not sure I understand what you proposed. If it is about
CAs computing 1's complement checksum for TCP/UDP packets,
there is no change for pkts on the wire and the whole checksum
offload can be hidden by implementations. There is nothing that
we need to do.

If you're proposing "checksum bypass", i.e., skipping the
1's complement checksum because the link layer is reliable
(i.e. the IB ICRC is stronger than the inet 1's complement
checksum therefore there is no need for the latter), I don't
know if this is all that feasible.

Could you give a more precise description of what you're
proposing to add to the WG's charter?

Jerry

> Date: Mon, 08 Jul 2002 19:55:33 +0200
> From: Yaron Haviv <[email-masked]>
> Subject: RE: [Ipoverib] Yokohama IETF
> To: "'Michael Krause'" <[email-masked]>, [email-masked]
> Cc: [email-masked]
> X-MIMEOLE: Produced By Microsoft MimeOLE V5.50.4807.1700
> Content-transfer-encoding: 7BIT
> Importance: Normal
> X-Priority: 3 (Normal)
> X-MSMail-priority: Normal
> Content-Transfer-Encoding: 7BIT
> X-Mailman-Version: 1.0
> List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
> X-BeenThere: [email-masked]
> 
> Mike,
> 
> If you look at my proposal in the original email you will see that:
> 1. The router in this case will verify the checksum for incoming (e.g. from
> Ethernet) and will keep the original value, in this case if it is routed
> through other I/O's there is no problem, since it is a completely valid
> TCP/IP packet with the right checksum value in place
> 2. For outgoing packets, during the connection setup the router will tell
> the node it will do the checksum for it (only if he is sure he can cover all
> the routing cases) than the node will send the packets to the router with
> only the pseudo header checksum, the router will fill the checksum
> 
> So in that case it does solve the problem since there won't be any illegal
> TCP/IP packets going anywhere other than it is supposed to
> We are focusing on the router example, however it could hold for other cases
> too
> 
> Yaron
> 
> -----Original Message-----
> From: Michael Krause [mailto:[email-masked]]
> Sent: Monday, July 08, 2002 6:31 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: RE: [Ipoverib] Yokohama IETF
> 
> At 06:44 PM 7/8/2002 +0200, Yaron Haviv wrote:
> >Mike,
> >You have referred only to one scenario: TCA
> >IPoIB as far as I know is the only standard for routing IP traffic in IB
> >As such it is totally legitimate to build a router that would route IP
> >traffic (e.g. from Ethernet) to IB based on IPoIB
> >RNDIS has certain usage model, it is a L2 solution, it works as an I/O
> >channel owned by its host, unlike routers
> >Some people may choose to have an RNDIS or a like IOC some might choose
> >otherwise for various reasons
> >
> >My proposal helps to increase performance, without going through too much
> >hassle It is not intended to be generic as I mentioned, just negotiated at
> >connection initiation phase You may choose not to use it; it works in a
> >way that it breaks nothing
> 
> At connection initiation one does not know whether the packet is to be
> routed via a different I/O through the endnode so such negotiation does not
> solve this problem.
> 
> >This whole email was to response on Jerry's WG scope email
> >I don't see why we can't add this offload as optional part of the standard,
> >just like RC & QPN, which has much less impact on performance
> 
> 
> 
> >you mention I haven't solved anything, what is there to solve with such an
> >approach ?
> >I don't see the reason for opposing to such a capability that has such
> >impact on performance
> >And could be implemented quite easily (much easier than doing it at the
> HCA)
> 
> Within a HCA it is just a 1's complement with minimal software fix-up
> within the host just like it is for Ethernet today.  The data is
> communicated as part of the completion event so there is nothing expensive
> to do here - it is quite trivial and I expect to see it added to most HCA
> as IB ramps to volume.  Thus for any QP type being used for IP
> communication, one can have this available without requiring anything from
> a standards group such as this.
> 
> Mike
> 
> 
> >Yaron
> >
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]]On Behalf Of
> >Michael Krause
> >Sent: Monday, July 08, 2002 3:30 PM
> >To: Yaron Haviv
> >Cc: [email-masked]'
> >Subject: RE: [Ipoverib] Yokohama IETF
> >
> >At 09:25 PM 7/7/2002 +0300, Yaron Haviv wrote:
> > >Mike, Jerry
> > >
> > >I understand that there are situations where checksum offloading is
> > >difficult to achieve in the generic sense and has potential problems
> > >However there is more than one case where it could be used and will bring
> > >much value e.g. in case of gateways, routers, xCA's supporting HW offload
> > >Since in RC the connection parameters can be negotiated, we can have
> > >offload when, and in the way that it could be verified we shouldn't block
> > >the cases where it could be easily supported from doing so  I think
> > >relaying on the HCA to do the work for us is a mistake, it wont happen in
> > >the near future, it also requires Verbs changes to tell the HW which QP
> > >should support offload, and an interface should be determined to tell the
> > >apps of errors, etc'
> >
> >It is not possible to achieve in a generic sense else we would have done it
> >years ago when cluster interconnects were first developed.  Nothing to date
> >has changed that capability.
> >
> >If a TCA acting as an IP terminus point implements checksum off-load this
> >is great but that has no impact on IP over IB which does not deal with such
> >a situation.
> >
> >It does not require Verbs changes unless you want it standard from the IBTA
> >and that is most likely not worthwhile to pursue at this time.
> >
> > >Just as an example an Ethernet TCA can verify the checksum for Ethernet
> > >packets if ok pass, if not discard, in any case leave the original
> > >checksum values in case the traffic will find it self routed to somewhere
> > >not aware of the offload on the other side when it negotiated that it
> > >supports checksum offload it can get the packet without the checksum and
> > >update the correct checksum in the outgoing Ethernet packets  This
> > >scenario is not limited to TCA's only, it could be used in any case where
> > >routed packets get the right treatment, or not routed at all, and this
> > >could be negotiated during the RC establishment, where by default the
> > >Checksum is always generated
> >
> >If people want to define an Ethernet TCA wire protocol that provides such
> >services, see RNDIS or create your own as RNDIS can only be used on one
> >OS.  However, the creation of such a wire protocol is outside of the scope
> >of IP over IB, IMO.
> >
> > >What holes do you see in that ??
> > >As you know most OS's support that feature today, so it doesn't involve
> > >changes to the TCP/IP stack Checksum offloading is critical especially in
> > >kernel apps that don't do an additional copy, we don't want to limit
> > >InfiniBand and make it perform less than Ethernet with a simple NIC
> (which
> > >has offload features), so where we can lets do use it
> >
> >You have not solved really anything with the above that has not been
> >considered before and determined to be a special before.
> >
> >Mike
> >
> >
> >_______________________________________________
> >IPoverIB mailing list
> >[email-masked]
> >https://www1.ietf.org/mailman/listinfo/ipoverib
> 
> 
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 12 Jul 2002 14:13:35 -0700 (PDT)

> proposing to add to the WG's charter?

The point that Yaron is making is that he wants to avoid having to do
the checksum if possible. This would be 'checksum bypass' as you term
it. There is no chekcsum carried on the wire as a result - in other
words this is a no-checksum option.

Here is a try at how it could work:

        The two peers have determined that both support IPoIB-RC mode.
        Now, at the device level they setup an RC connection between
        them. 

        At the time of the RC connection setup - when the CM REQ/REP
        messages are exchanged - the no-checksum option may be
        exchanged in private data field of the CM packet. It is a
        threeway handshake at the IB level.

        If accepted by the peers the IP interface reflects the option
        to the IP layer as 'checksum done in hardware' exactly as is
        the case with checksum off-loaded NICs. The difference is that
        in the IPoIB-RC-no-checksum case the on-wire packet will not
        include the checksum whereas on a 'checksum offload NIC' the
        checksum is actually inserted into the packet. 

        Since the option was negotiated at RC setup time, the two
        endpoints are very clear about the choice and if one of them
        chooses to forward the packets then it *must insert* the
        checksum before forwarding the packet unless that too is
        traversing on an IPoIB-RC-no-checksum-option link. The
        checksum-insertion happens below the IP layer. The
        pseudo-header checksum is included by the source even when
        using this option.

        But, if agreeable, does this apply to RC only ? I think not.
        Any mode that involves a link level negotiation (IPoIB-QPN,
        IPoIB-RC, IPoIB-UC etc.) and appears as a point to point link
        to IP can have such an option.

	I would like to explore this suggestion somewhat more. 

        Since the on-wire packet won't be carrying the checksum we
        need to decide on who gets the responsibility to ensure
        correct interoperation with non-IPoIB endpoints. It could be,
        as noted earlier, that accepting the option implies that one
        also accepts inserting the checksum when forwarding.

        For example, in the draft I have submitted
        (draft-kashyap-ipoib-advanced-capabilities-00.txt) the service
        IDs are derived from the upper layer ports( we can use a port
        of 0 to indicate the protocol itself).

        So, a host can associate an RC connection for a particular
        service (TCP/UDP port), transport protocol (e.g. separate RC
        for all UDP and a separate one for all TCP) or IP itself based
        on the ServiceID used.

        A host can therefore decide that RC connections corresponding
        to TCP connections terminating on itself can use the
        no-checksum option but not other RC connections. 
        This does imply that there may be multiple RC connections -
        the ones that carry the data terminating on the host accept
        this option but the RC connection that carries cross-subnet
        data doesn't.

        Since RC supports large MTUs (upto 2^31 bytes) one can even
        use UDP with this option without the worry of checksums across
        fragments when packets have to be forwarded. At the same time
        some might find it prudent to use the no-checksum option only
        for communication terminating on the receiver i.e. not support
        this option when forwarding.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: Thomas Narten <[email-masked]>
Date: Sat, 13 Jul 2002 19:55:51 -0400

The TCP/UDP checksums are end-to-end, between the TCP/UDP modules on
the sending and receiving nodes.

The IETF has never allowed disabling of such checksums as an
optimization (despite requests at various times to do so). There is
negative history here, for example, Sun's disabling of UDP checksums
for NFS (the theory being Ethernet's CRC was better anyway) resulted
in bad experience with corrupted data. Folks figured out that (say)
bridging ethernet via SLIP was useful. Oops.

In practice, it's very hard to guarantee that the actual path taken by
a packet has acceptable integrity properties. Packets can also be
damaged inside a forwarding device where CRCs are being recomputed,
but there is a period where there is no CRC.

This strikes me as an optimization attempt that needs very strong
justification that

1) it's a significant performance gain
2) it can be made to work reliably (without too much implementation
   complexity) in ALL cases.
3) stills post:  provide a true end-to-end check, as opposed to a series of
   hop-by-hop checks

Thomas

---

From: Erik Nordmark <[email-masked]>
Date: Sun, 14 Jul 2002 16:13:51 +0200 (CEST)

An issue in addition to the ones Thomas raised is that
checksum bypass doesn't interact well with other protocols.
For instance, we have Mobile IP which allows a node to move from its
home link and have a home agent take care of tunneling packets
to the current location of the node. If the connection is created
while the node is attached to the IB fabric, what would happen when the node
moves away from the IB fabric and have packets tunneled to it? 

Thus this type of checksum bypass is not robust in all dimensions.

   Erik

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 15 Jul 2002 10:25:21 -0700 (PDT)

Thomas,

> the sending and receiving nodes.

        That is true and that model must hold - end-to-end between the
        peer upper layer modules.

        The reason I have attempted to explore Yaron's suggetion is
        that IB uses a 32 bit CRC that is inserted at the source HCA
        and checked at the destination HCA. The intermediate IB
        switches/routers do not modify this CRC. There is also a hop
        by hop (on physical links between IB switches/routers).

        So there is end to end protection (within an IB fabric)
        exactly as checksum offload checksums in use
        today are inserted at the NIC and checked at the destination
        NIC, ignoring the pseudo-header checksum for the moment. The
        protection in this case is provided by an alternative method
        (CRC-32) instead of internet checksum. This CRC-32 is over the
        entire IB payload (TCP/UDP and IP) rather than just the
        TCP/UDP data though. One would still insert the pseudo-header
        checksum at the source always.

        So within an IB fabric things should be fine. On the face of
        it, it seems workable and thus my interest in exploring it. 
	
        If it doesn't work out because of some unworkable or
        cumbersome issues then we can drop this suggestion of avoiding
        the internet checksum over IB. Note that this suggesion
        doesn't apply to the IPoIB (over UD) transmission that we have
        been standardising. The dropping of the checksum has been
        suggested when doing IPoIB over IB's reliable connected mode
        only. 

        The real problems come up when we consider forwarding of
        packets onto non-IB links or IB links that don't provide this
        feature. Here I think IP fragmentation poses the biggest
        problem. 

                - Does the receiver reassemble all the packets before
                inserting the TCP/UDP chekcsum ?

                - Does the source always calculate checksum when
                fragmenting and the receiver (bridge/router) then
                doesn't bother with fragments in this regard?

                - IPoIB over RC can support large MTUs (2^31) so
                always use an MTU as large as the upper layer packet
                (64K for without window scaling for example). Thus the
                receiver will naturally receive the full packet and
                can then calculate the chekcsum over it when
                forwarding. 

        Even without fragmentation the receiver, as a result of
        agreeing to do 'no chekcsum' over the IB link, must also agree
        to insert the checksum when forwarding.

        My question is whether the above is workable or not ? In some
        cases it is simple - the receiver, when transmitting the
        packet over a non-IB media, sends the packet over a 'checksum
        offload' NIC which adds in the checksum. There might be other
        issues/situations that need to be considered.

> bridging ethernet via SLIP was useful. Oops.

        The idea would be that the bridge understands that the next
        hop is not-IB and hence it inserts the internet checksum. The
        CRC-32 protection will be active from the IB source to the
        bridge. 

        This is the reason why the communication has to be over a
        point to point link - that way the bridge knows that packets
        received over that source need checksum to be inserted.

> 1) it's a significant performance gain

        The gain for a host is same as with TCP checksum offload to
        the ethernet NIC.

>    complexity) in ALL cases.

        I agree that the complexity should be all in IB modules. 

> 3) stills post: provide a true end-to-end check, as opposed to a
series of hop-by-hop checks

	IB provides end to end CRC within the IB fabric.

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 15 Jul 2002 11:33:10 -0700 (PDT)

> moves away from the IB fabric and have packets tunneled to it? 

I am not sure if that is an issue since the suggestion is on
the link's capability, unless I have not understood your
comment. When transmitting over the IB link (with agreement in
place to no do the checksum) the checksum need not be done but
when communicating over an interface that does not provide
checksum offload/no-checksum option the checksum is always
inserted. 

This does bring up another issue though - who checks the
checksum when a packet is received from a source beyond the IB
fabric? 

It may be that IB node(A) and the router interface on IB side
(Ra) have the 'safe' IB link. The router does receive a packet
from another source(Z). The packet has the TCP checksum. The
Router sends it over the A-Ra link. 

The problem is 'how does A know that it must check the
checksum or not'? It needs to distinguish the packet from the
regular communication over the 'safe' link - if it has to
check it then we have lost the gain.

Do we ask the router to do the check????? same as mentioned
earlier that the router must insert the checksum when
forwarding ? Again one can get add an offload NIC to the
router on the receiving side and get this done on the card but
that is not a general purpose solution.

So, it does seem to be a good solution as long as one is
within a single IB fabric. When going beyond the problem
becomes that of finding a method of inserting the checksum and
checking the checksum.

The question then is whether the following is agreeable:

	when the two endpoints setup the link level connection
	and agree on not doing the internet checksum they also
	agree that when forwarding packets they verify the
	checksum on incoming packets and insert the checksum
	on outgoing packets.

	Some setups may be able to simply put a checksum
	offload NIC at the router to check/insert the
	checksums. But that is not a general purpose
	solution. 

Vivek

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 16 Jul 2002 01:48:36 +0200 (CEST)

> checksum offload/no-checksum option the checksum is always

You didn't understand my comment.

The sending and receiving TCP would need to agree (negotitate) that no checksum
is generated by the sender. Even if you can reliably always detect that  two
TCP peers are on a path that doesn't include anything beyond the CRC protected
path (which I don't think you can), you need a robust mechanism to deal with
the case when this property changes in the middle of a connection. While TCP
SYNs could potentially be extended for the initial negotiation, there is no
existing mechanism to renegotiate this in the middle of a TCP connection. In
summary, this would very very complex and IMHO still not robust.

> checksum when a packet is received from a source beyond the IB

See RFC 528 :-) The checksum is end-to-end.

  Erik

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 15 Jul 2002 20:45:44 -0700 (PDT)

> You didn't understand my comment.

        No. Let me make an attempt a better description and hopefull
        that will carify the matter. See below.

> is generated by the sender. 

No, not the sending and receiving TCP. But the source on IPoIB and the
edge router on the IPoIB subnet. If both the endpoints are on IPoIB
subnet then both the TCP systems have a direct IPoIB-RC link between
them. 

Yaron has not asked for TCP across the internet to ignore the checksum
but just within the IPoIB subnet. So he is only asking for the
checksum not to be carried over the IPoIB-RC link. 

I have attempted to look at how it could be done. It does seem
possible within the IPoIB subnet without much problem. 

Then I have attempted to see how the setup would interact with the
rest of the network. That is where the problems come up.

Even if you can reliably always detect that  two
> the case when this property changes in the middle of a connection. 

That is why I have repeatedly commented that this path will have to be
a 'point-to-point' i.e. a single hop between two endpoints. It doesn't
have to be the two TCP endpoints. Let me explain :

        A is a host and so is Z. R is a router. A and R are in the IB
        fabric. A sends a packet to Z which is somewhere in the
        internet with the access out of the IPoIB subnet being R.
        Assume that there is a checksum offload engine on the adapter
        exactly as is supported on many Ethernet NICs nowadays (so am
        not talking about no-checksum option just now).

        Now, A sends the packet to Z; it finds that it must first send
        the packet to R. The TCP stack in A has not inserted any
        checksum in the the TCP packet other than the calculation for
        the pseudo-header.  

        The packet is directed out to R at the link layer and the HCA
        inserts the checksum before sending it out.

        Now, if we consider the no-checksum option - the IP stack
        still doesn't change. No checksum is inserted, only
        pseudo-header checksum is inserted.

        The IP layer has determined Z is the destination, it also has
        determined that Z supports IPoIB-RC. So an RC connection is
        setup between A and Z - or one was setup at some earlier time.
        At the RC level, when the RC connection was setup the two ends
        had negotiated and agreed on 'no-checksum' option.

        Now the packet goes out from the HCA over this RC link
        directly to R. There is nothing between them.

        Now R knows that it received the packet on RC link from A and
        it has to forward it out some interface X. R therefore, must
        now insert the checksum into the packet.

        It may be that X is an ethernet NIC supporting chekcsum
        offload - and in this case R doesn't have to do anything but
        forward and the checksum gets added in.

        When R receives a packet from Z it can just forward it to A
        and then A can verify the checksum. But then the attempt on
        offload that Yaron is after will be lost or at least half
        lost. So R will have to calculate the checksum on the packet
        received from Z before sending it to A.

        So I was pointing out that if this suggestion is to work the
        above two issues, R checking and inserting checksum, must be
        acceptable to the WG.

	Thus the question is :
		 are the above conditions acceptable ?

        Now let us assume that this property changed in the middle of
        the connection ie. the IPoIB-RC connection was brought down. I
        can't think of any way the IPoIB-RC connection loses this
        property otherwise. Or, the route to Z changed. 

        In any case now that the TCP packet is to be sent to Z, A
        looks up the route and finds the associated interface. If the
        interface is IPoIB-RC with this option things work as before
        but if it is a different interface the checksum is inserted
        into the packet.

        The TCP connection is unaffected. This is exactly the same
        scenario with an ethernet offload engine NIC.

While TCP
> See RFC 528 :-) The checksum is end-to-end.

        You misunderstood what I meant. I hope the above description
        has clarified that.

        So my position is simply that 
		i) here is a suggestion to the group (from Yaron)
	       ii) analyse it and see if it seems workable. 
              iii) having done that exercise point out the cases that
              maybe of concern.

        If anything one can say that, based on the above example, R
        will act as the TCP checksum offload engine and send the
        packet to A. So the question on who checks the checksum was
        basically, whether R checks it or whether A checks it. Note
        the condition is that there is no intermediate node between R
        and A and that the link between them has a CRC that is checked
        across intermediate InfiniBand switches (below IP layer).

        I have not considered this suggestion to be equivalent to the
        IPoIB-RC link being same as the I/O bus and the remote IB node
        being equivalent to TCP checksum offload NIC because the
        suggestion also talks of a TCP/IP connection between two
        hosts. 

Vivek

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 16 Jul 2002 18:18:09 -0700 (PDT)

Erik,

	Yaron's Request:
                When sending IP frames over InfiniBand's reliable
                connected mode allow the two ends of the link to skip
                calculating the TCP/UDP checksum.

        My initial response was to reject the suggestion but then I
        realised that he has a point since InfiniBand's reliable
        connected mode is not ATM or Ethernet or any of the earlier
        technologies that couldn't get away with such an action.

        So, I decided to look into how it could be implemented and
        what problems it would face. I have described that - where it
        could be used and the problems in interaction with other
        links.  

        The CRC in InfiniBand is *source HCA to destination HCA if
        both the endpoints are in the same InfiniBand fabric*. The CRC
        is across intermediate InfiniBand switches/routers in the IB
        fabric. 

        Therefore, if an RC connection is setup between two systems
        the two ends are protected by a CRC from the HCA to HCA. This
        is no different than checksum-offload NICs since they too
        calculate the checksum in hardware and not in the IP stack in
        the host. The only difference is that the internet checksum is
        replaced by the CRC which is certainly end-to-end(as used in
        checksum offloaded NICs) in this case.

        So, it looks like a useable idea within an IPoIB subnet. See
        my previous mail to see how the implementation does not effect
        the TCP/IP stack in the host at all.

Problems:

        But problems crop up when the packet has to be sent beyond the
        IPoIB subnet. The RC connection is only upto the router. The
        packet cannot go beyond the router with the checksum not
        included. Therefore the checksum must be inserted somewhere.

        Similarly, the packet received by the router to be sent to the
        host at the other end of the RC connection, must be checked
        for the validity of the checksum.

        So, I asked the question of where can this be done? Is it
        acceptable to put a condition for 'no-checksum' option that
        implies that the router must check insert the checksum? The
        checksum cannot be inserted at the host since that would
        defeat the purpose of Yaron's request.

        Please note that the RC connection between the router and the
        host on the IPoIB subnet is reliable(guaranteed by the link
        layer), CRC protected through out the IB fabric, and
        point-to-point. 

        In earlier mails I also pointed out that the insertion of
        checksum is not that simple if there is fragmentation.

        A bridge (e.g. to ethernet) would also require to setup a
        similar InfiniBand reliable connection to talk RC and so the
        consideration wrt checksums is no different than for the
        router. 

So, there is no suggestion that TCP checksum is not end to end or that
checksum needs to be dropped when talking across the internet. The
suggestion is only dropping the checksum within the IPoIB subnet over
the IPoIB-RC link.

Basically, the case of no-checksum wholly within an IPoIB subnet is
attractive, for example a cluster in the IB fabric. It is a safe
optimisation *within the IB fabric*.

But going across the internet is another matter as I have pointed out
in the summary above and earlier mails. I share the concern wrt the
conditions put on the 'agreeing' router i.e. when doing IPoIB over RC
with no-checksum option, the router checks and inserts checksums.

It is just that the proposal shouldn't be rejected because something
similar was also suggested by IP over ATM (I've been told that ATM has
no CRC across intermediate ATM switches) or under the wrong impression
that the proposal wants to drop TCP checksums across the internet.
Hence my effort in explaining/analysing what exactly is implied by the
request. 

I hope this helps in clarifying some points in this discussion.

Vivek

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 17 Jul 2002 06:34:09 +0200 (CEST)

> Hence my effort in explaining/analysing what exactly is implied by the

AAL5 has a e2e CRC. IP runs over AAL5 over ATM.

But my point was not about specifics; it was merely a historical observation
that over the last few decades many folks have argued that their (feature rich)
link-layer provides checksums/CRCs thus the TCP checksum is superflous when
the path only includes that link. None of those suggestions have resulted in
IETF protocols that enable turning off the TCP checksum. You can make what you
want from this historical note.

One more point. The suggestion was to add a work item to the charter of the
form
	TCP checksum bypass (or whatever term was used)
I think the suggested work item should be renamed to
	The working group should revisit the end-to-end
	checksum part of the Internet architecture and see whether and were
	it makes sense to supress it and only rely on link layer mechanisms.

That way the whole IETF community can have a discussion 
1) whether the IETF should undertake work in this area and if so,
2) whether the ipoib WG is the place to do that work.

   Erik

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 17 Jul 2002 13:26:45 -0700 (PDT)

> want from this historical note.

I have not participated in earlier discussions on this topic in
other WGs but I can see at least two flaws in the 'no-checksum on the
link' argument when considering communication beyond the local
subnet. 

  - dependence on an edge device (router/bridge) to verify the checksum
        * with the possibility of it introducing an error which is
        then not verified by the receiving host on the 'protected
        link' or the host across the network.

        Actually a similar flaw exists in checksum-offload NIC based
        solutions which do not verify/insert the checksum in the IP
        stack of the host. In this case because of the reliable nature
        of IB RC link, the link is being equated to the bus transfer.
        But the possibility of error gets a little amplified because
        the nature of the edge device and data transfer.

  - difficulty with inserting the checksum when there are fragments.
    The edge device will have to collect all before it can
    introduce/verify the checksum.

>   it makes sense to supress it and only rely on link layer mechanisms.

        :). Only if one can say - when the communication is fully
        contained within the 'protected link' based local subnet. I
        don't know if one can.

Vivek

---

From: Michael Krause <[email-masked]>
Date: Sun, 21 Jul 2002 07:43:17 -0700

>                 inserting the TCP/UDP chekcsum ?

None trivial and at high-speeds would require potentially very large 
quantities of card-resident memory or an implicit scatter into host memory 
while maintaining the current checksum state until all packets are received.

>                 can then calculate the chekcsum over it when

A message still goes through SAR at the IB level requiring one to maintain 
this state across multiple packets and for an arbitrary amount of time 
since the packets may be interleaved across multiple messages.

Note: For QoS purposes, one will have to established multiple RC 
connections between endnodes perhaps using diffserv code points so that 
head-of-line blocking from these large messages does not interfere with 
other traffic.  Same also applies to the current UD proposal though the OS 
would naturally interleave messages as it does today with Ethernet.

>         to insert the checksum when forwarding.

Within the forwarding element, someone must provide local data integrity 
protection on these intermediate buffers so that nothing can be corrupted 
without detection, e.g. ECC memory.

>         issues/situations that need to be considered.

I've never had a problem with an IB/Ethernet TCA providing checksum 
off-load services since it is the real layer 2 from the IP 
perspective.  This is certainly workable from an IP perspective as well as 
an IB perspective.  Even for an endnodes acting as a router, it works just 
fine since the Ethernet provided the checksum before the packet left the 
endnode from an IP perspective.  The issue is what if the leaving the 
source does not have the checksum and someone has to calculated on a middle 
box - in this case, the checksum is no longer end-to-end and one must have 
more guarantees on data integrity within each component along the way to 
establish any trust model from a customer perspective.  It may seem like 
splitting hairs here since the models are not that radically different from 
one another, but is something to consider.

>         IB provides end to end CRC within the IB fabric.

That is not what was being asked since the bridge is technically hop-by-hop 
with the entire IB fabric being viewed as a single point-to-point 
connection (akin to PCI) to an IB/Ethernet module.  It would be simpler to 
just view this as a TCA issue and not an IP over IB issue and define how 
one communicates / translates an IP over IB packet to a TCA at the edge of 
the IB fabric.  If this were the case, then one could disable checksum 
generation except at the edge as long as all intermediate components 
provided strong memory protection in all components (e.g. ECC) which is 
somewhat hard to do at this time since lots of people still do not ship ECC 
host memory let alone always abide by local I/O data integrity 
requirements.  Not necessarily any worse than how the Internet operates 
today in terms of exposure to silent data corruption but something to consider.

Mike