Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

From: Yaron Haviv <[email-masked]>
Date: Thu, 13 Dec 2001 23:35:43 +0200

Just trying to finalize my arguments 

As for having 4 bits or 4 byes ethertype:
Comparing 4 bits takes exactly the same resources as 4 bytes
If the header size is constant the header processing is not complicated,
when the next header offset is variable (in case of IP options, LLC, etc')
you need to parse the packet
Its not the case here you just start the IP header processing in offset 4,
after you checked the first 4 bytes
Putting the 4 bits in pieces of the IB header is less natural
We have done several NIC drivers in different OS's, including an IPoIB
driver
So I could tell you based on our experience it is not a problem

As for the demuxing 
1. The network driver doesn't process the ARP's, the OS above it does
2. You separate to different queues when A) you want to save context switch,
copies or b) when you want to avoid going over the queue and look for your
specific information, this is not the case here we are in the same context,
the ARP messages are sporadic and could wait in the queue for a while until
I will get to them 

As I mentioned before you are looking for optimizations in the wrong place,
there are many other places where we could optimize in future work 

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Vivek Kashyap
Sent: Thursday, December 13, 2001 1:41 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: Re: [Ipoverib] IPoIB architecture/encapsulation

Yaron,
        I am afraid we have sidetracked. The comment about performance
        was different. The point I was making was that generally you
        would get better performance if you had less to do. It doesn't
        mean that one cannot by specialisation mitigate the negatives
        of a more complicated task.

          delete/add header vs. no such work. I like no work.

        Here is the problem and the reasoning:

        The network stack doesn't change you merely add a network
        interface driver. This driver has the task of determining the
        upper layer to call. It can call it directly or it can slot
        the packet in some type based queue. The interface driver will
        determine the type from the packet stream. It can do so in
        many ways. One of the ways is by using the protocol signature.
        Another is by way of 'ethertype' and a third is by mandating a
        QP per protocol.

        The use of 'ethertype' is not universal. A choice by IPoIB to
        use ethertype does not mean 'protocol X' over IB will use the
        same encapsulation and therefore you cannot rely on
        'ethertype' differentiating your packets.

        So the 'ethertype' is of use in differentiating IPoIB
        protocols only (IPv4, IPv6 and ARP). For this task we have two
        other methods that require no change in the on-wire
        encapsulation: use the protocol signatures (4 bits in the
        begining) and use separate QPs per protocol. Of the two : use
        protocol signatures doesn't require any on-wire change,
        doesn't require the driver to strip headers, doesn't require
        the TCP layer to keep space for the header. That is why it is
        preferable. 

        In essence the protocol signature solution gives us the same
        returns with the additional advantage of :

                - no need to reflect it on the wire
                - no removal/addition of headers

Vivek

>
> Vivek,
>
> I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on
mem access as well as CPU cycles
> My assumption is that OSV's will not change the way their TCP/IP stack is
built and separate ARP's from IP's, they will just add IPoIB as another net
interface driver
> The demultiplexing doesn't add much value since it is done in the same
kernel context, there is no urgency in dealing with the ARP's, there aren't
many ARP's, it also relates to the stack work (ARP response might release
pending IP packets)
>
> it looks to me like the 4 bit option is not the straightforward approach
>
> If you want to improve performance significantly, without affecting the
stack you should do things similar to what the new NIC's do:
> -  Offload checksum (avoid cache trashing)
> -  Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
> -  Suppress Interrupts
> -  Improve memory management
> These things could be done with RC/RD and using some IB tricks
>
> Yaron
>
> Original Message:
> -----------------
> From: Vivek Kashyap [email-masked]
> Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
> To: [email-masked], [email-masked]
> Subject: Re: [Ipoverib] IPoIB architecture/encapsulation
>
>
> Yaron,
>
> >
> > Few questions & suggestions
> >
> > I'm trying to understand way not to take the straightforward approach
for
> > the ethrtype field
> > And just add that before the IP header in the payload area, if alignment
is
> > needed so align it
>
>         Since we already filter the recepients by way of QP to one
>         (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
>         work is the same whether you see 4 bits or 32 bits. But it is a
>         lot less with 4 bits.
>
>         The interface driver will distinguish the types and call the
>         upper layer accordingly.
>
>                 It is easier with 4 bits than with 4 bytes. Some people
>                 have mentioned performance - I would say that if you
>                 have less to do you are more performant. With 4 bits
>                 you have less to do. Secondly, if you are in
>                 'performance' mode switching in hardware then it is
>                 even better since you don't have to strip the header
>                 but merely have to pass the buffer to the relevant
>                 upper layer directly.
>
>         The interface driver does not have to do the followig when
>         using protocol signatures:
>
>     - It doen't have to add a header when sending
>     - It doesn't have to receive header when receiving
>
>   These tasks are required for if one invents an 'ethertype'.
>
>         The upper layer doesn't have to keep space for the header, the
>         ethertype, when allocating data for sending if the protocol
>         signatures are to be used.
>
>         I don't see any gain if an artificial header is included to
>         distinguish protocols in IPoIB. I see more work. We already
>         know, by definition, it is IP or ARP. Just test which one it
>         is by looking at the first 4 bits.
>
>
> >
> > I might be missing something here so forgive me if I do
> > It looks to me like TCP stacks are used to get a packet with everything
> > including Ethernet header, it is not like we are doing full header/data
> > separation that could help the stack
> > On the worst case if the TCP implementation has problems with it we
could
> > have 2 data segments in the receiving side, one for the first word,
second
> > for the rest
>
> Things are generally done in simpler ways. The IP stack gets the IP
> packet. The lower headers are stripped away. If you don't have the
> any header to strip in your interface driver all the better.
>
> >
> > Another thing that was mentioned in the recent emails was that IPoIB is
> > defined just for UD
>
>         If you look at the beginning of this thread (the mail that
>         started it all) I have discussed capability flags and how
>         RC/RD/UC connections can be setup and work interoperably with
>         UD only implementations. IPoIB over UD is the default and
>         madatory.
>
>
> Vivek
>
> _______________________________________________
> IPoverIB mailing list
> [email-masked]
> http://www.ietf.org/mailman/listinfo/ipoverib
>
> --------------------------------------------------------------------
> mail2web - Check your email from the web at
> http://mail2web.com/ .
>
>

--
Vivek Kashyap
IBM
[email-masked]
[email-masked]
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 13 Dec 2001 16:23:24 -0800 (PST)

> you need to parse the packet

        Is it less complicated than not having the header at all ?
        Sure one can work with the header but why do something when
        there is no need for it? 

        In my previous mail I explained the advantages of protocol
        signature over the 'ethertype'. Those advantages are not
        overcome by your argument. All you are saying is that one can
        make the receive processing in the presence of 4 byte
        'ethertype' as good as the 'protocol signature' no-header
        processing.  

> Putting the 4 bits in pieces of the IB header is less natural

        I am afraid I have lost you. There is no putting of 4 bits of
        pieces in IB header. The 4 bits I refer to are already part of
        the packet.

        The IP packet starts with the IP version (a 4 bit field) and
        the ARP encapsulation starts with hardware type (a 16 bit
        value). These values are different in the first 4 bits as I
        explained in an earlier mail about the values. Hence the
        information is already present in the packet and there is no
        need to add a header. As noted earlier we are disambiguating
        between 2 protocols only.

> We have done several NIC drivers in different OS's, including an
IPoIB > driver > So I could tell you based on our experience it is not
a problem

        Anything can be made to work. I only object to have to make
        something work that has not been shown to bring anything to
        the table.

> 1. The network driver doesn't process the ARP's, the OS above it does

        Yes, but the driver or some layer just above it demultiplexes
        the streams.

> the ARP messages are sporadic and could wait in the queue for a while until
	
        Sure one can do as you say. Put them all in one queue and then
        process them on the type. Why would one need to invent a
        header for that? All one has to do is to look at the protocol
        signature. 

> there are many other places where we could optimize in future work 

        I am pointing out that if there is no advantage from more
        work then less work is always better than more work.

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)