Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

From: "H.K. Jerry Chu" <[email-masked]>
Date: Sat, 12 Jan 2002 16:51:31 -0800 (PST)

Sorry I am late to the party (was stuck on something else most of
Dec.), but like to make some comments regarding EtherType.

While I don't buy the performance arguement, I sympathize with
Vivek on his feeling toward EtherType. Architecturally it is ugly
to force one into the payload. The right approach would have the
framing layer carrying a field identifying the type of data being
encapsulated. But BTH wasn't designed for encapusulation purpose.
That is supposed to be the job and the only purpose of raw datagrams. 
Unfortunately due to various reasons the latter has been considered
by many as not fit for the job of IPoIB encapsulation.

Aside from the architectural correctness, I only see two valid concerns
against not including EtherType:

1. Compatibility issue - how many existing stack implementations
expect EtherType, or part of framing header to be present in the
datastream? Although the driver can always fabricate a pseudo header
if one is needed but is missing, it may incur some performance
penalty since it won't be in-line.

2. Not including Ethertype forbids one from sharing the same QP with
other protocols outside of IPv4/ARP/IPv6. Is this a legitimate concern?
Note that this concerns only applies unicast. For multicast, by
embedding a special IPoIB signature in MGIDs for IPv4/ARP/IPv6 use
only, demuxing can be based on MGID so sharing QPs won't be an issue.

Before the two questions are fully understood, I'm leaning slightly
toward including the EtherType, just to be on the safe side.

Jerry

>From: Vivek Kashyap <[email-masked]>
>Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>To: [email-masked] (Yaron Haviv)
>Date: Thu, 13 Dec 2001 16:23:24 -0800 (PST)
>Cc: [email-masked]
>> Just trying to finalize my arguments 
>> 
>> As for having 4 bits or 4 byes ethertype:
>> Comparing 4 bits takes exactly the same resources as 4 bytes
>> If the header size is constant the header processing is not complicated,
>> when the next header offset is variable (in case of IP options, LLC, etc')
>> you need to parse the packet
>
>        Is it less complicated than not having the header at all ?
>        Sure one can work with the header but why do something when
>        there is no need for it? 
>
>        In my previous mail I explained the advantages of protocol
>        signature over the 'ethertype'. Those advantages are not
>        overcome by your argument. All you are saying is that one can
>        make the receive processing in the presence of 4 byte
>        'ethertype' as good as the 'protocol signature' no-header
>        processing.  
>
>> Its not the case here you just start the IP header processing in offset 4,
>> after you checked the first 4 bytes
>> Putting the 4 bits in pieces of the IB header is less natural
>
>        I am afraid I have lost you. There is no putting of 4 bits of
>        pieces in IB header. The 4 bits I refer to are already part of
>        the packet.
>
>        The IP packet starts with the IP version (a 4 bit field) and
>        the ARP encapsulation starts with hardware type (a 16 bit
>        value). These values are different in the first 4 bits as I
>        explained in an earlier mail about the values. Hence the
>        information is already present in the packet and there is no
>        need to add a header. As noted earlier we are disambiguating
>        between 2 protocols only.
>
>
>> We have done several NIC drivers in different OS's, including an
>IPoIB > driver > So I could tell you based on our experience it is not
>a problem
>
>        Anything can be made to work. I only object to have to make
>        something work that has not been shown to bring anything to
>        the table.
>
>> 
>> As for the demuxing 
>> 1. The network driver doesn't process the ARP's, the OS above it does
>
>        Yes, but the driver or some layer just above it demultiplexes
>        the streams.
>
>> 2. You separate to different queues when A) you want to save context switch,
>> copies or b) when you want to avoid going over the queue and look for your
>> specific information, this is not the case here we are in the same context,
>> the ARP messages are sporadic and could wait in the queue for a while until
>> I will get to them 
>	
>        Sure one can do as you say. Put them all in one queue and then
>        process them on the type. Why would one need to invent a
>        header for that? All one has to do is to look at the protocol
>        signature. 
>
>> 
>> As I mentioned before you are looking for optimizations in the wrong place,
>> there are many other places where we could optimize in future work 
>
>        I am pointing out that if there is no advantage from more
>        work then less work is always better than more work.
>
>Vivek
>
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]On Behalf Of
>> Vivek Kashyap
>> Sent: Thursday, December 13, 2001 1:41 AM
>> To: [email-masked]
>> Cc: [email-masked]
>> Subject: Re: Re: [Ipoverib] IPoIB architecture/encapsulation
>> 
>> Yaron,
>>         I am afraid we have sidetracked. The comment about performance
>>         was different. The point I was making was that generally you
>>         would get better performance if you had less to do. It doesn't
>>         mean that one cannot by specialisation mitigate the negatives
>>         of a more complicated task.
>> 
>>           delete/add header vs. no such work. I like no work.
>> 
>>         Here is the problem and the reasoning:
>> 
>>         The network stack doesn't change you merely add a network
>>         interface driver. This driver has the task of determining the
>>         upper layer to call. It can call it directly or it can slot
>>         the packet in some type based queue. The interface driver will
>>         determine the type from the packet stream. It can do so in
>>         many ways. One of the ways is by using the protocol signature.
>>         Another is by way of 'ethertype' and a third is by mandating a
>>         QP per protocol.
>> 
>>         The use of 'ethertype' is not universal. A choice by IPoIB to
>>         use ethertype does not mean 'protocol X' over IB will use the
>>         same encapsulation and therefore you cannot rely on
>>         'ethertype' differentiating your packets.
>> 
>>         So the 'ethertype' is of use in differentiating IPoIB
>>         protocols only (IPv4, IPv6 and ARP). For this task we have two
>>         other methods that require no change in the on-wire
>>         encapsulation: use the protocol signatures (4 bits in the
>>         begining) and use separate QPs per protocol. Of the two : use
>>         protocol signatures doesn't require any on-wire change,
>>         doesn't require the driver to strip headers, doesn't require
>>         the TCP layer to keep space for the header. That is why it is
>>         preferable. 
>> 
>>         In essence the protocol signature solution gives us the same
>>         returns with the additional advantage of :
>> 
>>                 - no need to reflect it on the wire
>>                 - no removal/addition of headers
>>        
>> 
>> Vivek
>> 
>> 
>> >
>> > Vivek,
>> >
>> > I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on
>> mem access as well as CPU cycles
>> > My assumption is that OSV's will not change the way their TCP/IP stack is
>> built and separate ARP's from IP's, they will just add IPoIB as another net
>> interface driver
>> > The demultiplexing doesn't add much value since it is done in the same
>> kernel context, there is no urgency in dealing with the ARP's, there aren't
>> many ARP's, it also relates to the stack work (ARP response might release
>> pending IP packets)
>> >
>> > it looks to me like the 4 bit option is not the straightforward approach
>> >
>> > If you want to improve performance significantly, without affecting the
>> stack you should do things similar to what the new NIC's do:
>> > -  Offload checksum (avoid cache trashing)
>> > -  Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
>> > -  Suppress Interrupts
>> > -  Improve memory management
>> > These things could be done with RC/RD and using some IB tricks
>> >
>> > Yaron
>> >
>> > Original Message:
>> > -----------------
>> > From: Vivek Kashyap [email-masked]
>> > Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
>> > To: [email-masked], [email-masked]
>> > Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>> >
>> >
>> > Yaron,
>> >
>> > >
>> > > Few questions & suggestions
>> > >
>> > > I'm trying to understand way not to take the straightforward approach
>> for
>> > > the ethrtype field
>> > > And just add that before the IP header in the payload area, if alignment
>> is
>> > > needed so align it
>> >
>> >         Since we already filter the recepients by way of QP to one
>> >         (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
>> >         work is the same whether you see 4 bits or 32 bits. But it is a
>> >         lot less with 4 bits.
>> >
>> >         The interface driver will distinguish the types and call the
>> >         upper layer accordingly.
>> >
>> >                 It is easier with 4 bits than with 4 bytes. Some people
>> >                 have mentioned performance - I would say that if you
>> >                 have less to do you are more performant. With 4 bits
>> >                 you have less to do. Secondly, if you are in
>> >                 'performance' mode switching in hardware then it is
>> >                 even better since you don't have to strip the header
>> >                 but merely have to pass the buffer to the relevant
>> >                 upper layer directly.
>> >
>> >         The interface driver does not have to do the followig when
>> >         using protocol signatures:
>> >
>> >     - It doen't have to add a header when sending
>> >     - It doesn't have to receive header when receiving
>> >
>> >   These tasks are required for if one invents an 'ethertype'.
>> >
>> >         The upper layer doesn't have to keep space for the header, the
>> >         ethertype, when allocating data for sending if the protocol
>> >         signatures are to be used.
>> >
>> >         I don't see any gain if an artificial header is included to
>> >         distinguish protocols in IPoIB. I see more work. We already
>> >         know, by definition, it is IP or ARP. Just test which one it
>> >         is by looking at the first 4 bits.
>> >
>> >
>> > >
>> > > I might be missing something here so forgive me if I do
>> > > It looks to me like TCP stacks are used to get a packet with everything
>> > > including Ethernet header, it is not like we are doing full header/data
>> > > separation that could help the stack
>> > > On the worst case if the TCP implementation has problems with it we
>> could
>> > > have 2 data segments in the receiving side, one for the first word,
>> second
>> > > for the rest
>> >
>> > Things are generally done in simpler ways. The IP stack gets the IP
>> > packet. The lower headers are stripped away. If you don't have the
>> > any header to strip in your interface driver all the better.
>> >
>> > >
>> > > Another thing that was mentioned in the recent emails was that IPoIB is
>> > > defined just for UD
>> >
>> >         If you look at the beginning of this thread (the mail that
>> >         started it all) I have discussed capability flags and how
>> >         RC/RD/UC connections can be setup and work interoperably with
>> >         UD only implementations. IPoIB over UD is the default and
>> >         madatory.
>> >
>> >
>> > Vivek
>> >
>> > _______________________________________________
>> > IPoverIB mailing list
>> > [email-masked]
>> > http://www.ietf.org/mailman/listinfo/ipoverib
>> >
>> > --------------------------------------------------------------------
>> > mail2web - Check your email from the web at
>> > http://mail2web.com/ .
>> >
>> >
>> 
>> 
>> --
>> Vivek Kashyap
>> IBM
>> [email-masked]
>> [email-masked]
>> 503 578 3422 (o)
>> 
>> _______________________________________________
>> IPoverIB mailing list
>> [email-masked]
>> http://www.ietf.org/mailman/listinfo/ipoverib
>> 
>
>
>-- 
>Vivek Kashyap
>Linux Technology Center, IBM
>[email-masked] 
>[email-masked] 
>503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 14 Jan 2002 11:55:01 -0800 (PST)

I couldn't have put it better wrt why Ethertype is not needed: it is
 ugly. It really buys us nothing:

	* It does not buy us differentiation

                a) If one has a QP, by definition one knows what to
                   expect. There is no need for any type field.

                b) Ethertype does not help with differentiating with
                   other protocols. They are under no obligation to
                   follow any header defined by IPoIB. Are we
                   considering any headers that might be proposed by
                   SDP/SRP or any other encapsulation in IB ? No.
                   Similarly there nothing binding the rest of the IB
                   world to follow the rules set by IPoIB.

        We expect a small set of packets at a QP. It is easy to
        differentiate between these based on the signatures of the
        draft. The only packet we expect on an IPoIB QP are : IPv4/v6
        and ARP. These in themselves are easily distinguishable.

> penalty since it won't be in-line.

[ I thought you said you don't buy any performance argument :). ]

        Does an 'classical IP over ATM' or 'ITX' return ethertype ?
        Secondly, if an implementation needs a type the driver can
        return it to the upper layers in any form one desires - as an
        argument to a function, as a type in the buffer header,
        another appended header. It can surely be done very
        efficiently. 

        Anyway, that is an internal issue of the implementation. Why
        impose it on implemenations that don't need/want it ? The ones
        that wnat it can add the header internally. Why put it on the
        wire where it is not needed ?

> only, demuxing can be based on MGID so sharing QPs won't be an issue.

        As noted before one will need everybody to follow the same
        rules as IPoIB to work this way. That is unlikely to happen.
        Only IBTA can dictate universal rules for all of IB.

> toward including the EtherType, just to be on the safe side.

	Another method out of this logjam is allow both - hold on :).

        In IP over ethernet one sets up the interface with SNAP (like
        Ethertype) and NOSNAP(no ethertype). Let us follow that. If
        set SNAP/Ethertype the interface will send and expect
        Ethertype header. If not it will work without it. This allows
        an endpoint to use whatever option they prefer.

        In another year or two we will know which is preferred and
        will live with that. 

Vivek