Subject: RE: Re: [Ipoverib] IPoIB architecture/encapsulation

From: "[email-masked]" <[email-masked]>
Date: Wed, 12 Dec 2001 17:03:03 -0500

Vivek,

I generally agrees with Mike, comparing 4 bit vs. 4 bytes has no impact on mem access as well as CPU cycles
My assumption is that OSV’s will not change the way their TCP/IP stack is built and separate ARP’s from IP’s, they will just add IPoIB as another net interface driver
The demultiplexing doesn’t add much value since it is done in the same kernel context, there is no urgency in dealing with the ARP’s, there aren’t many ARP’s, it also relates to the stack work (ARP response might release pending IP packets)

it looks to me like the 4 bit option is not the straightforward approach 

If you want to improve performance significantly, without affecting the stack you should do things similar to what the new NIC’s do:
-	Offload checksum (avoid cache trashing)
-	Increase MTU, MSS (avoid UDP fragmentation, TCP segmentation)
-	Suppress Interrupts 
-	Improve memory management 
These things could be done with RC/RD and using some IB tricks 

Yaron

Original Message:
-----------------
From: Vivek Kashyap [email-masked]
Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)
To: [email-masked], [email-masked]
Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

Yaron,

> 
> Few questions & suggestions
> 
> I'm trying to understand way not to take the straightforward approach for
> the ethrtype field
> And just add that before the IP header in the payload area, if alignment is
> needed so align it

        Since we already filter the recepients by way of QP to one
        (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
        work is the same whether you see 4 bits or 32 bits. But it is a
        lot less with 4 bits. 

        The interface driver will distinguish the types and call the
        upper layer accordingly.

                It is easier with 4 bits than with 4 bytes. Some people
                have mentioned performance - I would say that if you
                have less to do you are more performant. With 4 bits
                you have less to do. Secondly, if you are in
                'performance' mode switching in hardware then it is
                even better since you don't have to strip the header
                but merely have to pass the buffer to the relevant
                upper layer directly.

        The interface driver does not have to do the followig when
        using protocol signatures:

		- It doen't have to add a header when sending
		- It doesn't have to receive header when receiving

	These tasks are required for if one invents an 'ethertype'.

        The upper layer doesn't have to keep space for the header, the
        ethertype, when allocating data for sending if the protocol
        signatures are to be used.

        I don't see any gain if an artificial header is included to
        distinguish protocols in IPoIB. I see more work. We already
        know, by definition, it is IP or ARP. Just test which one it
        is by looking at the first 4 bits.

> 
> I might be missing something here so forgive me if I do
> It looks to me like TCP stacks are used to get a packet with everything
> including Ethernet header, it is not like we are doing full header/data
> separation that could help the stack
> On the worst case if the TCP implementation has problems with it we could
> have 2 data segments in the receiving side, one for the first word, second
> for the rest

Things are generally done in simpler ways. The IP stack gets the IP
packet. The lower headers are stripped away. If you don't have the
any header to strip in your interface driver all the better.

> 
> Another thing that was mentioned in the recent emails was that IPoIB is
> defined just for UD

        If you look at the beginning of this thread (the mail that
        started it all) I have discussed capability flags and how
        RC/RD/UC connections can be setup and work interoperably with
        UD only implementations. IPoIB over UD is the default and
        madatory. 

Vivek

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 12 Dec 2001 15:40:49 -0800 (PST)

Yaron,
        I am afraid we have sidetracked. The comment about performance
        was different. The point I was making was that generally you
        would get better performance if you had less to do. It doesn't
        mean that one cannot by specialisation mitigate the negatives
        of a more complicated task.

	  delete/add header vs. no such work. I like no work. 

        Here is the problem and the reasoning:

        The network stack doesn't change you merely add a network
        interface driver. This driver has the task of determining the
        upper layer to call. It can call it directly or it can slot
        the packet in some type based queue. The interface driver will
        determine the type from the packet stream. It can do so in
        many ways. One of the ways is by using the protocol signature.
        Another is by way of 'ethertype' and a third is by mandating a
        QP per protocol.

        The use of 'ethertype' is not universal. A choice by IPoIB to
        use ethertype does not mean 'protocol X' over IB will use the
        same encapsulation and therefore you cannot rely on
        'ethertype' differentiating your packets. 

        So the 'ethertype' is of use in differentiating IPoIB
        protocols only (IPv4, IPv6 and ARP). For this task we have two
        other methods that require no change in the on-wire
        encapsulation: use the protocol signatures (4 bits in the
        begining) and use separate QPs per protocol. Of the two : use
        protocol signatures doesn't require any on-wire change,
        doesn't require the driver to strip headers, doesn't require
        the TCP layer to keep space for the header. That is why it is
        preferable.  

        In essence the protocol signature solution gives us the same
        returns with the additional advantage of :

		- no need to reflect it on the wire
		- no removal/addition of headers

Vivek

-- 
Vivek Kashyap
IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: "Jim Fleming" <[email-masked]>
Date: Wed, 12 Dec 2001 20:39:23 -0600

----- Original Message -----
From: <[email-masked]>

> My assumption is that OSV's will not change the way their TCP/IP stack is built and separate ARP's from IP's, they will just add
IPoIB as another net interface driver.
----

InfiniBand is a different animal. It has its own architecture (IBA) and trade groups
supporting it.  http://www.infinibandta.org
While it is true that InfiniBand may be handicapped by making it look like just
another network interface, that is not likely to change the fact that the IBA architecture
can move forward without the IAB, IESG, IETF, IANA, ISOC, ICANN and IEEE.

Performance will be the realm where InfiniBand shines. Plug-and-play will also be a
key factor. Retrofitting InfiniBand with 1970's packet technology does not seem to
be a winning direction for InfiniBand.

Jim Fleming
http://www.IPv8.info
IPv16....One Better !!

---

From: Ashwin Kamath <[email-masked]>
Date: Thu, 13 Dec 2001 11:44:17 -0600

Vivek,
  Some questions,
1. Is there no possibility that a protocol other than IPv4/IPV6/ARP be 
added to use the IPOIB resources?
2. If an extra protocol needs to be added then would it be much better 
that the protocol definition be assisted by something like the 
ether-type field so as to resolve de-muxing ambiguities? We cannot solve 
a third protocols ambiguity problem if it uses the IpoIb resources, more 
so because such a protocol does not exist. But looks like we can 
certainly help the definition of such a protocol, if it needs to exist 
in the future, by suggesting that it use the ether-type field. If we use 
only the first 4 bits of the header, we throttle any such possibility. 
Is this desirable?

-Ashwin.

[email-masked] wrote:

> My assumption is that OSV's will not change the way their TCP/IP stack is built and separate ARP's from IP's, they will just add IPoIB as another net interface driver
> The demultiplexing doesn't add much value since it is done in the same kernel context, there is no urgency in dealing with the ARP's, there aren't many ARP's, it also relates to the stack work (ARP response might release pending IP packets)
> 
> If you want to improve performance significantly, without affecting the stack you should do things similar to what the new NIC's do:
> X-te: Wed, 12 Dec 2001 12:27:20 -0800 (PST)

--

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 13 Dec 2001 11:24:42 -0800 (PST)

Ashwin,

> added to use the IPOIB resources?

        You may say that IB can carry more protocols but that doesn't
        make such protocols part of IPoIB.

> only the first 4 bits of the header, we throttle any such possibility. 

        The real way to solve this is to move the solution to the IB
        layer. The packet identifier in the BTH of IB headers solves
        the problem at the IB layer. 

        IPoIB being a higher layer cannot solve an intrinsic problem
        of IB the lower layer.

Vivek