Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

From: "Vivek Kashyap" <[email-masked]>
Date: Tue, 11 Dec 2001 11:28:30 -0800

Roy,

 I misread it and assumed wrongly. Yes I agree, use of immediate data looks
like a valid solution.

We have three solutions to avoid 'ethertype' in payload:

     1) use protocol 'signatures'

     2) immediate data

     3) packet identifier in IB packets (needs IB upgrade being pursued in
IBTA)

We need to discuss it a little more. Wouldn't 'protocol signatures' work ?
Are there any special requirements with IB's immediate data ? Would it be
ok to state that once packet identifiers are  available we can move forward
to 3) in an interoperable way ?

At present I favor 1) with a later followup to 3). You favor 2). In my view
all are valid solutions. I'll be able to talk in more detail after reading
the IB spec on immediate data.

Vivek

PS: We are doing IPoIB. IB has many transports. The default choice is UD.
It is also clear that no one favors using Raw mode. However many have shown
desire for advantages offered by other transport modes. Please see earlier
mails and the beginning of this thread for some views on other IB modes.
Note: IPoIB over UD is mandatory and the default.

--
Vivek Kashyap
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                    "Larsen, Roy                                                                                  
                    K"                   To:     Vivek Kashyap/Beaverton/IBM@IBMUS                                
                    <roy.k.larsen@       cc:     [email-masked]                                                
                    intel.com>           Subject:     RE: [Ipoverib] IPoIB architecture/encapsulation             
                                                                                                                  
                    12/11/2001                                                                                    
                    10:28 AM                                                                                      

Vivek,

You misread what I wrote, which was:

>I realize immediate data could not be
>used with raw data packets, but capabilities outside of UD for IPoverIB
are
>beyond the scope of what we're trying to get drafted here.

Immediate data is available for all message types EXCEPT raw packets.
(Refer to table 60 on page 388 of the IBA spec.)  Using raw packets as a
transport mechanism is outside the scope of the IPoverIB spec as is using
RC, UC, and RD services.  Given this, can I assume you support immediate
data for conveying the ethertype?

Roy

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: "Yaron Haviv" <[email-masked]>
Date: Tue, 11 Dec 2001 23:25:34 +0200

Few questions & suggestions

I'm trying to understand way not to take the straightforward approach for
the ethrtype field
And just add that before the IP header in the payload area, if alignment is
needed so align it

I might be missing something here so forgive me if I do
It looks to me like TCP stacks are used to get a packet with everything
including Ethernet header, it is not like we are doing full header/data
separation that could help the stack
On the worst case if the TCP implementation has problems with it we could
have 2 data segments in the receiving side, one for the first word, second
for the rest

Another thing that was mentioned in the recent emails was that IPoIB is
defined just for UD
In the discussion in LWG few months ago I recall few capability bits in one
of the suggestions
I personally think we should not preclude the use of RC/RD for IPoIB unicast
traffic and leave it as an option
This could be used in future enhancements to increase MTU to "Jumbo Frames",
add help avoiding the Checksum calculations

In the near future RC might assist in supporting profile A chips that
doesn't have large MTU's, and will be deployed in environments where they
provide enough value (I would have hoped to have just Profile B chips,
however they are here, with us, and probably for some time)
For some of those limited chips that also doesn't support Multicast we could
implement a multicast server that will either convert the traffic to
multicast or send it to the multiple clients, it might also proxy the ARP's
and answer based on his internal table & SA information
To be clear I don't like it !!! (see ATM), but it could do the work as an
interim solution that will help accelerate IB adoption with the given HW
I assume that someone building a multimedia solution relying on UDP
multicast will probably wont use such HW, however it might be sufficient for
ARP's, DHCP's, etc'

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Vivek Kashyap
Sent: Tuesday, December 11, 2001 9:29 PM
To: [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Roy,

     2) immediate data

Vivek

                    "Larsen, Roy
                    K"                   To:     Vivek
Kashyap/Beaverton/IBM@IBMUS
                    <roy.k.larsen@       cc:     [email-masked]
                    intel.com>           Subject:     RE: [Ipoverib] IPoIB
architecture/encapsulation

                    12/11/2001
                    10:28 AM

Vivek,

are

Roy

---

From: "Bill Strahm" <[email-masked]>
Date: Tue, 11 Dec 2001 23:16:01 -0800

There are varying implementations for what the IP stack looks for.  Most of
them that I am aware are looking for a byte stream that starts with the
first byte of the IP header.  That is not to say if you did
ipstream[-ethhdrsize] you wouldn't accidentally find the beginning of the
Ethernet header.  However the stacks that I have worked with DO NOT do
things like this (and I would personally shoot any developer that did <G>)

We are currently trying to define IP over UD because it is easiest and most
supported.  Beyond this I am willing to look at proposals to do any of the
other IB transports... Are you going to write the draft ???

By the way... What is UDP Multicast ??? I am assuming you mean IP multicast
and the fact that UDP is running above it is just a coincidence...

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Yaron Haviv
Sent: Tuesday, December 11, 2001 1:26 PM
To: 'Vivek Kashyap'; [email-masked]
Subject: RE: [Ipoverib] IPoIB architecture/encapsulation

Yaron

-----Original Message-----

Roy,

     2) immediate data

Vivek

                    "Larsen, Roy
                    K"                   To:     Vivek

                    12/11/2001
                    10:28 AM

Vivek,

are

Roy

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 12 Dec 2001 12:27:20 -0800 (PST)

Yaron,

> And just add that before the IP header in the payload area, if alignment is

        Since we already filter the recepients by way of QP to one
        (IPv6) or two (IPv4/ARP) why go to this trouble? The type of
        work is the same whether you see 4 bits or 32 bits. But it is a
        lot less with 4 bits. 

        The interface driver will distinguish the types and call the
        upper layer accordingly.

                It is easier with 4 bits than with 4 bytes. Some people
                have mentioned performance - I would say that if you
                have less to do you are more performant. With 4 bits
                you have less to do. Secondly, if you are in
                'performance' mode switching in hardware then it is
                even better since you don't have to strip the header
                but merely have to pass the buffer to the relevant
                upper layer directly.

        The interface driver does not have to do the followig when
        using protocol signatures:

		- It doen't have to add a header when sending
		- It doesn't have to receive header when receiving

	These tasks are required for if one invents an 'ethertype'.

        The upper layer doesn't have to keep space for the header, the
        ethertype, when allocating data for sending if the protocol
        signatures are to be used.

        I don't see any gain if an artificial header is included to
        distinguish protocols in IPoIB. I see more work. We already
        know, by definition, it is IP or ARP. Just test which one it
        is by looking at the first 4 bits.

> have 2 data segments in the receiving side, one for the first word, second

Things are generally done in simpler ways. The IP stack gets the IP
packet. The lower headers are stripped away. If you don't have the
any header to strip in your interface driver all the better.

> Another thing that was mentioned in the recent emails was that IPoIB is

        If you look at the beginning of this thread (the mail that
        started it all) I have discussed capability flags and how
        RC/RD/UC connections can be setup and work interoperably with
        UD only implementations. IPoIB over UD is the default and
        madatory. 

Vivek

---

From: Michael Krause <[email-masked]>
Date: Wed, 12 Dec 2001 13:04:42 -0800

>         work is the same whether you see 4 bits or 32 bits. But it is a

Variety of ways this may be implemented involving one or multiple QP based 
on hardware and software filters.

>                 but merely have to pass the buffer to the relevant

All of the normal headers including this 4 byte field with all fit on the 
same cache line so the performance savings are unclear.  Also, if 
performance is critical, then one would be performing header / data split 
using some form of a local scatter list which again would be independent of 
the 4 byte in-line ethertype.

>                 - It doen't have to add a header when sending

It is a store operation which depending upon driver structure / design may 
be a cached object so nothing special need be done.

>                 - It doesn't have to receive header when receiving

See above. It really may not matter per se. BTW, there may also be benefit 
in seeing this header as one might create some interesting software-based 
dispatching policies for subsequent packet processing.

>         signatures are to be used.

Noise as the cache line is already touched.

>         is by looking at the first 4 bits.

Argument applies to both approaches and could be reduced to deltas are 
mouse nuts and the ubiquity deployment possibility being the dominant 
decision point.

>any header to strip in your interface driver all the better.

I slice these just like Ethernet headers today and given this is UD, I have 
to slice a bit anyway.

>         UD only implementations. IPoIB over UD is the default and

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Wed, 12 Dec 2001 13:41:49 -0800 (PST)

> on hardware and software filters.

        Exactly. And if you already have the QP filtering for you then
        why do you need to identify it yet again in a pseudo-header.

        There is only one QP associated with the IP stream in the
        default UD mode. You know exactly what you are getting.

> the 4 byte in-line ethertype.

        The point is that you don't have to worry about a header. This
        header isn't providing any additional information. All the
        information is already included in the packet and the QP
        context. No one has mentioned any advantage 'ethertype' has
        over the 4 bit approach. If both are the same as you say but
        one requires adding/stripping of headers then why not go with
        one that doesn't require this adding/stripping ?

> be a cached object so nothing special need be done.

        it may be a cached object but you do need to create space for
        it in your data packet. If you don't have a header you neither
        have to create such a cached object nor have to create space
        for it. You don't have to deal with it period. 

        It is ok to invent a header if there is a clear cut advantage.
        There is none. You get the same benefit by leveraging the QP's
        filtering and the protocol signature.

> dispatching policies for subsequent packet processing.

        Why do you need to see the header for that ? All you need to
        know is the type. The proposed header has the type : value x
        is IP; value y is ARP. It gets the same x =4 and y = 0 for IP
        and ARP respectively without having to deal with additional
        headers. 

        Any software based policy is internal to the implemenation and
        need not show up on wire. You can easily implement any
        software based policy internal to the system.

        An interface strips the header and calls the relevant upper
        layer. If one wants to use a cached object for inventive
        packet processing internally one may do so. Prepend it to the
        data packet received and send it up. This choice of some
        shouldn't be reflected in the wire protocol.

> Noise as the cache line is already touched.

        Cache line is not the issue. It is keeping track of an
        unnecessary header when you can avoid it. 

> mouse nuts and the ubiquity deployment possibility being the dominant 

        I don't know what you mean. The task is a simple test. Then
        why not just test instead of going about inventing a header
        which provides the same information as already present.

Vivek

-- 
Vivek Kashyap
IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)