Subject: RE: [ipoverib] resolving the Ethertype issue

From: "Jim Pinkerton" <[email-masked]>
Date: Mon, 28 Jan 2002 07:29:10 -0800

I'm in the "resolve it quick" camp as well. I don't strongly believe one
way or the other, but would like to directly address Bill's question. As
Yaron states, the copy on current networking stacks occurs between the
transition from the protocol stack (SO layer), typically in the kernel,
to the application process, typically in user space. 

The transition from the device driver to the IP layer typically is done
by simply moving a pointer to the next header (i.e. no copy is
involved). For example, say the packet is:

+----------+   <- ptr handed to device driver
| LRH      |
+----------+
| GRH      |
+----------+
| BTH      |
+----------+
| DETH     |
+----------+   <- ptr handed to IP layer
| IP hdr   |
+----------+
| TCP hdr  |
+----------+   <- ptr handed to SO (sockets) layer
| payload  |
|          |
//        //
|          |
+----------+

So as the packet is handed up the stack, the same buffer is used and the
pointer into the buffer is simply advanced to the next header as the
packet is handed up the stack. Of course IB has some unique issues
because of the completion queue, so the DLL portion of the above is over
simplified, but it does not materially change the discussion. Then at
the final SO (sockets) layer, this is where either a page flip or a copy
occurs (in today's implementations) into the application buffer. Another
model which can avoid a copy is if the application api supports the
protocol layer giving the app a pointer to the buffer (rather then the
other way around, which is what sockets requires). In any of those cases
though, for the copy issue it is orthogonal as to whether the ethertype
is present or not. The ethertype simply adds another few bytes to
increment the pointer to get to the IP header (i.e. no big deal). 

So in conclusion I don't believe the copy avoidance algorithm is
materially effected by whether an ethertype is included or not.

On the issue of eating QP - if the goal is now to use one QP each for
IPv4, IPv6, and ARP, I am concerned. QP from our perspective are not
cheap. They should be used primarily if there is a security issue or
performance issue, and preferably all traffic should be able to co-exist
on the same QP otherwise. For the case of IPv4 mixing with IPv6 mixing
with ARP, it seems like any performance or security win is marginal,
thus making it seem reasonable to ensure that all traffic can go on a
single QP if possible, possibly while enabling implementations that want
to put them on different QP to do so. 

If that is the goal, it appears as though the cleanest implementation is
to have the ethertype. If it is not there though, I agree that we can
hack it up by looking at the first nibble and detecting whether it is an
ARP, IPv4, or IPv6 packet. I'm not sure this would make it through the
IESG though. 

Jim

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]] 
Sent: Monday, January 28, 2002 1:14 AM
To: 'Bill Strahm'; [email-masked]; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

Bill,

No copy is/should be done between L2-L3, you just pass the buffer
pointer
(sk_buff in linux, m_buff in BSD)
The mandatory copy is between the kernel and user space (TCP/IP private
buffers to Application receive buffers)
If an application can work with the packets when they are in the TCP/IP
private buffers chains (without doing a receive(buf) op for example) so
it
could avoid the copy, usually it is not so simple, might work better for
kernel Apps
Another approach is to build TOE NIC's which place the data directly in
the
application receive buffers (where the private buffers are on the NIC,
no
magic)

The protocol demuxing relates to the execution process
The L2 look @ the protocol number, and calls the relevant protocol
function
(or soft IRQ) based on the value, this could also be done from L3 to L4

By the way adding more QP's doesn't solve any problem, it just requires
more
resources (I had a long email on that in the pass)

My take on the Ethertype issue is that who ever needs it when its not
provided could generate it for him self
Where who ever wants to ignore it when its available could also do that
This is why I don't think it is a critical decision if we do have it or
we
don't, since the raw information is provided
It is up to the group to see if someone feels (and could also explain)
that
the Ethertype provides him an Info he must have
And not just for future Ethernet over IB stuff since we deal with IP now

I personally could sleep well with the two decisions with/without, I
just
want to resolve it quick

Yaron

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of
Bill Strahm
Sent: Monday, January 28, 2002 10:24 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using
Ethertype
without incurring either an additional copy, or a page flip ?  The way I
see
it is you will have to install a software "protocol demuxer" above the
IB
layer to receive the IB frame, look at the first 4 bytes and then put it
on
the appropriate protocols receive buffer.  I am most interested in
listening
to peoples implementations that involve zero-copy going up the TCP
stack.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.
In
that case you CAN NOT determine if you are looking at a "EtherType" or
the
first 4 bytes of a payload therefore...
</Bill>
If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet
here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to
worry
about forcing all other encapsulations to include a 4 byte header in
their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle
upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>

Billhttps://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Koji takahara <[email-masked]>
Date: Tue, 29 Jan 2002 01:03:59 +0900

Bill,

> given use of Infiniband.  I thought that was the purpose of QPN.

I think IPoverIB standard should not specify how to use 
multiple QPNs.

I am not afraid of using multiple QPNs.
I want to use multiple QPNs for the purpose of load balancing.
I do not want to use multiple QPNs for the purpose of 
distiguishsing ethertypes. 

> I claim you will have to do this anyway
 
As Roy and Culley said, when using method (2), 
IPv4 and ARP must have special relation. 
I think it unclean.

As Yaron and Jim said, including ethertype at the head of 
payload does not have copy problem.

_______________________________________________________________
Koji Takahara  Email: [email-masked]
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted

---

From: Vivek Kashyap <[email-masked]>
Date: Mon, 28 Jan 2002 10:52:31 -0800 (PST)

There have been a few notes referring to expanding IPoverIB to other
protocols. That is not correct. We are implementing 'IP over
InfiniBand' and that is it. Any other protocol that is/has to be
carried over IB will have to define its own encapsulation. We cannot
force them to use this unnecessary header ? So let us keep this
discussion limited to IP over InfiniBand and that implies IPv4/ARP and
IPv6. 

Those that prefer a type to distinguish between InfiniBand's
protocols, if they are IBTA members, can look at a proposal in LWG
that discusses including a type in the IB headers. This type will
allow all IB protocols to follow the same rules. Any format discussed
or specified in this forum(ipoverib) is limited, by definition, to
IPoverIB and is not binding on any other protocol. Thus there is no
way to distinguish the packets based on a header that is followed only
by 2 protocols (IP and ARP) if another protocol's packets come to the
same QP. The real solution of this type can come from IBTA only.

Basically, the desire is to receive IP and ARP on the same QP. To
distinguish these at the driver level (which many don't seem to
consider peeking into the next layer) you want a type. Why should it
matter what the size of the type is? If the driver looks at the first
4 bits it can easily distinguish between the three types
of(IPv4/ARP/IPv6). 

We should NOT be including the Ethertype for the following reasons:

      * there is no way to distinguish the ethertype from 'data' of
      another protocol since it is not bound by the rules of IPoverIB
      Since we limit the protocols by the QP then we can distinguish
      using the first 4 bits.

      * type in IB headers will make it unnecessary anyway

      * the receiver is limited to three protocols
            IPv4/ARP/IPv6

      The driver considers the first four bits as the type. It doesn't
      care about the rest of the payload and its format. The values
      are very specific: 4/6/0 respectively. The only difference from
      using a 4 byte ethertype is that the driver will not move the
      pointer forward to skip over the ethertype but just pass the
      received buffer as is to the relevant upper layer.

      * it doesn't seem right that to provide for some implementations
      wanting to use existing hardcoded values the solution should be
      pushed onto the wire. Instead it is very easy,for example, for
      the receiver to recognise a value of 4 in the first 4 bits and
      return it as 0x800 to the relevant function. Other
      implementations that merely pass the packet to the relevant
      protocol's handler and do not need a type can also work easily.

As far as I know the type is to distinguish the packets and pass them
to the relevant upper layer. The driver skips over the type and the
upper layer works with the rest of the packet. The same happens very
easily without the ethertype as described above.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Koji takahara                                                                                                    
                      <[email-masked]        To:       [email-masked]                                                             
                      su.co.jp>                cc:                                                                                     
                      Sent by:                 Subject:  RE: [ipoverib] resolving the Ethertype issue                                  
                      ipoverib-admin@ie                                                                                                
                      tf.org                                                                                                           

                      01/27/2002 09:08                                                                                                 
                      PM                                                                                                               
                      Please respond to                                                                                                
                      koji                                                                                                             

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)