Subject: [ipoverib] resolving the Ethertype issue

From: Yaron Haviv <[email-masked]>
Date: Fri, 25 Jan 2002 19:46:21 +0200

As Arvind mentioned the Ethertype field issue is one of the last items to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the only
thing that Vivek proposal might not enable is doing Ethernet over IB which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like “it is
better”, etc'

Yaron

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Culley, Paul" <[email-masked]>
Date: Fri, 25 Jan 2002 15:51:42 -0600

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]]
Sent: Friday, January 25, 2002 2:36 PM
To: [email-masked]'; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

Although my feelings aren't strong enough to drag the reflector into a
never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated
as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<prc>
True, but at the moment, only two protocols are in our charter to
differentiate.  These could be easily identified by examining the first
nibble of the packet; IP has 4 or 6 in the first nibble, surly we can
chose some other value to id ARP.  Or you can use the following QP
proposal.  Or both, if the implementation wants both protocols on one
QP.
</prc> 

2.  QPNs do not provide a clean mechanism to do this because, at a
minimum,
both ARP responses and IPv4 frames must share the same QPN.
<prc>
 I don't see why this must be so.  I seems perfectly possible to include
an ARP return QP in the ARP request that would be used in the ARP
response.  The ARP response can, in turn include the IP QP for use by
the IP traffic.  If gratuitous ARP and ARP snooping are
allowed/supported, then the ARP request should also include the separate
IP QP.  This solves the problem for the two protocols that our group is
chartered to solve.
</prc>

3.  Requiring the IPoIB driver to peek into an upper-level protocol
frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<prc>
Whatever works...
</prc>

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for
IB
subnet routing issues.)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Friday, January 25, 2002 9:46 AM
To: [email-masked]
Subject: [ipoverib] resolving the Ethertype issue

As Arvind mentioned the Ethertype field issue is one of the last items
to
agree up on

Currently there are 2 proposals:
1.	(Vivek) no need for ethertype, we could see its an ARP from the
first
fields in the Packet (IP Ver)
2.	put the Ethertype field in the first few bytes of each message

I personally don't see it as a critical decision choosing 1 or 2, the
only
thing that Vivek proposal might not enable is doing Ethernet over IB
which
is not in the scope of this RFC anyway
I would tend to agree with Vivek for the simplicity reasons (not the
performance ones)

If any one has a firm reason why to put the Ethertype field in the
payload
please make your arguments
And it would help in settling this matter too
Please try to give real reasons in this case too, and not things like
“it is
better”, etc'

Yaron

---

From: "Larsen, Roy K" <[email-masked]>
Date: Fri, 25 Jan 2002 15:50:26 -0800

embedded <rkl>

-----Original Message-----
From: Culley, Paul [mailto:[email-masked]]
Sent: Friday, January 25, 2002 1:52 PM
To: Larsen, Roy K; [email-masked]; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

embedded...

Paul R. Culley
Compaq Sr. Fellow
281-514-5543

-----Original Message-----
From: Larsen, Roy K [mailto:[email-masked]]
Sent: Friday, January 25, 2002 2:36 PM
To: [email-masked]'; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

Although my feelings aren't strong enough to drag the reflector into a never
ending rat hole, I believe the IPoIB draft should specify a
protoco/"Ethertype" as the first field of the payload which is treated as an
IPoIB protocol specific field.  The reasons are as follows:

1.  Interesting OSs require the link driver to identify the destination
protocol of received frames.
<prc>
True, but at the moment, only two protocols are in our charter to
differentiate.  These could be easily identified by examining the first
nibble of the packet; IP has 4 or 6 in the first nibble, surly we can chose
some other value to id ARP.  Or you can use the following QP proposal.  Or
both, if the implementation wants both protocols on one QP.
</prc> 
<rkl>
Actually, there are three, ARP, IPv4, IPv6.  Each are treated as distinct
protocols in the OSs I'm interested in.  I've already expressed my view on
the integrity of peeking into protocol headers.
</rkl>

2.  QPNs do not provide a clean mechanism to do this because, at a minimum,
both ARP responses and IPv4 frames must share the same QPN.
<prc>
 I don't see why this must be so.  I seems perfectly possible to include an
ARP return QP in the ARP request that would be used in the ARP response.
The ARP response can, in turn include the IP QP for use by the IP traffic.
If gratuitous ARP and ARP snooping are allowed/supported, then the ARP
request should also include the separate IP QP.  This solves the problem for
the two protocols that our group is chartered to solve.
</prc>
<rkl>
Are you suggesting that a single request/response transaction should not
resolve the IPv4 endpoints on both sides?  If so, we need to explicitly
state that the QPN in the source hardware address field of the ARP request
is only valid for ARP responses and MUST NOT be assumed valid for IPv4
frames.  I assumed this was not the case since it would increase ARP traffic
by a factor of 2.  Surely a protocol type field is a much smaller price to
pay than requiring every node to initiate an ARP even for nodes it has
already been in ARP-replay contact with.  I'm not even sure my OSs can deal
with this..... 
</rkl>

3.  Requiring the IPoIB driver to peek into an upper-level protocol frame is
architecturally displeasing. (Read that as an euphemism for KLUDGE.)
<prc>
Whatever works...
</prc>
<rkl>
An engineering approach I've used in the past, but only when left with no
cleaner solution.
</rkl>

4.  Arguments that some day the IB header may contain such a field are
irrelevant in defining a protocol today.  (Same reasoning we've seen for IB
subnet routing issues.)

Roy

-----Original Message-----
From: Yaron Haviv [mailto:[email-masked]]
Sent: Friday, January 25, 2002 9:46 AM
To: [email-masked]
Subject: [ipoverib] resolving the Ethertype issue

Yaron

---

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 28 Jan 2002 11:08:49 -0800

My comments embedded with <rkl>

-----Original Message-----
From: Bill Strahm [mailto:[email-masked]]
Sent: Monday, January 28, 2002 12:24 AM
To: [email-masked]; [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

Comments inline...

I guess my big question is why are people afraid of allocating QPN for a
given use of Infiniband.  I thought that was the purpose of QPN.

Can someone tell me how they can redirect packets to the correct Layer 3
(with IB UD being a Layer 2 protocol for this discussion) using Ethertype
without incurring either an additional copy, or a page flip ?  The way I see
it is you will have to install a software "protocol demuxer" above the IB
layer to receive the IB frame, look at the first 4 bytes and then put it on
the appropriate protocols receive buffer.  I am most interested in listening
to peoples implementations that involve zero-copy going up the TCP stack.

Bill

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of Koji takahara
Sent: Sunday, January 27, 2002 9:08 PM
To: [email-masked]
Subject: RE: [ipoverib] resolving the Ethertype issue

I strongly agree with Roy.

I think the best method to identify the destination protocol is
the following.
(1) to specify a Ethertype as the first field of the payload
<Bill>
How do you propose handling the case where an Layer 3 (for IB that is)
protocol does not put a n-byte (I assume 4-byte) header in the message.  In
that case you CAN NOT determine if you are looking at a "EtherType" or the
first 4 bytes of a payload therefore...
</Bill>
<rkl>
I have seen this stated by others as well and I don't know what it means.
How would an arbitrary protocol choose the IPoIB QPN(s)?  I assume it would
be through their own address/neighbor resolution protocol which means it
would be under the control of the responding node and would presumably
followed a common link layer (from the OS point of view) protocol header.
So, if all Foo-over-IB encapsulation protocols used the same "link" header,
it would be very easy to integrate in to the interesting OSs network stack
with a common driver.  But that is not really the issue.  The issue is
protocol layering integrity. 
</rkl>

If we do not use method (1), we have to use one of the following
methods.
(2) to use different QPNs for different Ethertypes, or
<Bill>
I claim you will have to do this anyway
</Bill>
<rkl>
There is no need to do that at all. Our implementation will use a single QPN
for all encapsulation data because that is the most seamless way to
integrate into the demultiplexing interface of the OS.  But of course,
others are free to use more.
</rkl>
(3) to peek into an upper-level protocol frame in link layer
<Bill>
You consider looking at the first 4 bytes of the ULP acceptable, yet here
you claim it is unclean... which is it ???
</Bill>
I also think both (2) and (3) are not clean.

And I think we should not assume Ethertypes are only ARP, IPv4,
IPv6.
That assumption restricts the expantion of ther upper-layer of
the ipoverib link layer driver.
<Bill>
Correct.  I claim option 2 is the way to go.  Then you don't have to worry
about forcing all other encapsulations to include a 4 byte header in their
framing, I do not see a way of enforcing this anyway..

What are peoples objections to using a small number of QPN to handle upper
layer protocols on top of Infiniband.  My assumptions are QPN are very
similar to Ethertypes except they can be dynamic...
</Bill>
<rkl>
For ARP transaction efficiency, I believe APR and IPv4 must use the same
QPN.  I've started another thread to hopefully get consensus on this.
<rkl>

Billhttps://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "Larsen, Roy K" <[email-masked]>
Date: Mon, 28 Jan 2002 11:53:31 -0800

As I said in my first post, this issue is not worth dragging this into a
never ending rat hole for me.  So this will be my last public words on the
subject.

This is not an issue about enabling non-IPoverIB protocols.  The
encapsulation author has agreed that ARP and IPv4 MUST share the same QPN.
There are those of us who want to use a single link address for all IPoIB
traffic.  As a matter of protocol layering integrity, I support defining an
IPoIB encapsulation header that contains a destination protocol field.  I do
not know of a single widely used network protocol that does not include such
a destination specifier at every layer and, for the life of me, can't
understand resistance to one.  As Jim P. stated in a previous post, there
may be those in the standards community that will not indorse a protocol
that requires violating layering integrity.  Arguments that a proposed
change to the IBA will cover this are presumptuous and therefore irrelevant.

Yes, we can peek into the IP frames.  Yes we can peek into the ARP frame,
Yes, IP could peek into the TCP frame.  Yes, TCP could peek into user data.
It is just architecturally dishonest and without benefit.

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Monday, January 28, 2002 10:53 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ipoverib] resolving the Ethertype issue

There have been a few notes referring to expanding IPoverIB to other
protocols. That is not correct. We are implementing 'IP over
InfiniBand' and that is it. Any other protocol that is/has to be
carried over IB will have to define its own encapsulation. We cannot
force them to use this unnecessary header ? So let us keep this
discussion limited to IP over InfiniBand and that implies IPv4/ARP and
IPv6. 

Those that prefer a type to distinguish between InfiniBand's
protocols, if they are IBTA members, can look at a proposal in LWG
that discusses including a type in the IB headers. This type will
allow all IB protocols to follow the same rules. Any format discussed
or specified in this forum(ipoverib) is limited, by definition, to
IPoverIB and is not binding on any other protocol. Thus there is no
way to distinguish the packets based on a header that is followed only
by 2 protocols (IP and ARP) if another protocol's packets come to the
same QP. The real solution of this type can come from IBTA only.

Basically, the desire is to receive IP and ARP on the same QP. To
distinguish these at the driver level (which many don't seem to
consider peeking into the next layer) you want a type. Why should it
matter what the size of the type is? If the driver looks at the first
4 bits it can easily distinguish between the three types
of(IPv4/ARP/IPv6). 

We should NOT be including the Ethertype for the following reasons:

      * there is no way to distinguish the ethertype from 'data' of
      another protocol since it is not bound by the rules of IPoverIB
      Since we limit the protocols by the QP then we can distinguish
      using the first 4 bits.

      * type in IB headers will make it unnecessary anyway

      * the receiver is limited to three protocols
            IPv4/ARP/IPv6

      The driver considers the first four bits as the type. It doesn't
      care about the rest of the payload and its format. The values
      are very specific: 4/6/0 respectively. The only difference from
      using a 4 byte ethertype is that the driver will not move the
      pointer forward to skip over the ethertype but just pass the
      received buffer as is to the relevant upper layer.

      * it doesn't seem right that to provide for some implementations
      wanting to use existing hardcoded values the solution should be
      pushed onto the wire. Instead it is very easy,for example, for
      the receiver to recognise a value of 4 in the first 4 bits and
      return it as 0x800 to the relevant function. Other
      implementations that merely pass the packet to the relevant
      protocol's handler and do not need a type can also work easily.

As far as I know the type is to distinguish the packets and pass them
to the relevant upper layer. The driver skips over the type and the
upper layer works with the rest of the packet. The same happens very
easily without the ethertype as described above.

Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                      Koji takahara

                      <[email-masked]        To:       [email-masked]

                      su.co.jp>                cc:

                      Sent by:                 Subject:  RE: [ipoverib]
resolving the Ethertype issue                                  
                      ipoverib-admin@ie

                      tf.org

                      01/27/2002 09:08

                      PM

                      Please respond to

                      koji

_______________________________________________________________
Koji Takahara  Email: [email-masked]
Development Dept.5,  Infrastructure Systems division,
Computer Group, Fujitsu Limitted

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)