Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

From: "Ashwin Kamath" <[email-masked]>
Date: Wed, 6 Mar 2002 09:57:08 -0600

Sorry for the late reply

I don't think this is what I meant. How will an two implementations, one
using a single qp per type and another using one qp for all types
interoperate and work with each other. More specifically how will they
interoperate with advanced features using Reliable Connections? A reliable
connection is point to point, so if one point uses multiple queupairs for
different transports and another point uses a single queue pair for all
transports, these two points cannot interoperate. If they have to
interoperate, an ARP response must contain queue pair information for all
ethertypes that it supports. This is impractical. So having an ethertype
does not make interop much more attractive, and its addition is questionable
for advanced features.

The reason I called it ethernet over IB is because what you are proposing is
ipoib support the same functions as ethernet does today. If that is not the
goal of the WG, I think that any other addition must be justified by a.
interop, b. performance, c. consensus (probably in that order ). I was
trying to point out that adding ether-type does not help interop. I can't
see any other impact on performance either, question is do we have a
consensus to add the ether-type? (This paragraph is my view on the matter)

-Ashwin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of [email-masked]
Sent: Thursday, February 28, 2002 7:03 AM
To: [email-masked]
Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

>From: "Ashwin Kamath" <[email-masked]>
>Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>Date: Wed, 27 Feb 2002 22:59:30 -0600

Ashwin,

I try to write two examples (with/without Ethertype) using the following
 figure.

  node A                                        node B
+--------------+---+                      +---+--------------+
|IPv4/ARP/RARP |   |                      |   |IPv4/ARP/RARP |
+----+         | Y |                      | Y |          +---+
| X  |         |   |                      |   |          | X |
+----+---------+---+                      +---+----------+---+
|  ipoib           |                      |   ipoib          |
+-+-----+------+-+-+                      +-+-+--+-------+-+-+
  |     |        |  (c)EtherType = 0xd000   |    |         |
  |     |        +--------------------------+    |         |
  |     |                                        |         |
  |     |     (a)EtherType = 0x800/0x806/0x8035  |         |
  |     +----------------------------------------+         |
  |                                                        |
  |           (b)EtherType = 0xeeee (e.g.)                 |
  +--------------------------------------------------------+

>Akihiko,
> How would you propose implementations working with multiple queuepairs for
>different transports to interoperate with your implementation.

> Do you
>propose that everybody use the ether-type and a single queue pair for all
>ethernet traffic.

> Then we would be solving ethernet-over-IB and not ipoib,
>is this right?

>-Ashwin.

I don't understand why you say "ethernet traffic" and "ethernet-over-IB"
but I try to write.

# Sorry for only first broadcast packet send and receive processes examples.

-----------------------
["With Ethertype"
 An implemetation by a single QP per interface and multiple protocols per
QP]

1) At first, Master SM initializes ports in its IB-subnet and set them
   to ACTIVE.
2) In a node, a ipoib driver initializes itself.
  (create only one QP for the ipoib interface
   ,join the broadcast MCG after getting p_key (which is assigned to each
    IP-subnet or may be shared by both IP on X subnet and IP on ipoib
subnet.)
   ,get Q_Key from the MCG
   ,if need,modify Q_Key of QP (or destroy and re-create a QP with a new
Q_Key)
   ,attach the QP to HCA's MCG table on the node.
   ,post receive-buffers to HCA)

3-a) When a "protocol"(IPv4,ARP,RARP,X or Y) starts to use an ipoib
interface,
  ipoib gets Ethertype of the upper layer protocol.
3-b) When a "protocol"(IPv4,ARP,RARP) starts to use an X interface,
  X gets Ethertype of the upper layer protocol.

4) Here an only QP for every protocol (i.e. IPv4,ARP,RARP,X,Y) per ipoib
   interface is ready to send and receive packets.

-- "ARP on ipoib" --
5) For example,
   ARP on ipoib in node A reqeusts for ipoib to send an ARP REQUEST packet
   with Ethertype 0x806 on top of payload to the IPv4 broadcast MCG.

6-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. ARP packet) and
   sees that Ethertype in payload indicates ARP
    and passes this packet to ARP on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.

6-b) ARP on ipoib can get node A's IPv4 address on ipoib and MAC
     or invalidate node A's IPv4 address on ipoib IPaddr-MAC cache entry.

-- "ARP on X" --
7) ARP on X in node A reqeusts for X to send an ARP REQUEST packet to
   the IPv4 broadcast MCG,
   and X requests for ipoib to send a packet with Ethertype 0xeeee
   which encapsulates(encodes. e.g. add X's header) the ARP packet.

8-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. X packet) and
   sees that Ethertype in payload indicates X
    and passes this packet to X on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
8-b) X on ipoib decodes(e.g. strips X's header) X packet into ARP packet,
    and passes this packet to ARP on X.
8-c) ARP on X can get node A's IPv4 address on X and MAC
     or invalidate node A's IPv4 address on X IPaddr-MAC cache entry.

   Assumption:
      X can distinguish protocol of the packet is ARP. It is up to X.

-- "Y's own packet on ipoib" --
9) Y on ipoib in node A reqeusts for ipoib to send an Y's own packet
   with Ethertype 0xd000 to the IPv4 broadcast MCG.

10-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. Y packet) and
   sees that Ethertype in payload indicates Y
    and passes this packet to Y on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
10-b) Y on ipoib get node A's Y dependent info,

---------------------
["Without Ethertype"
 An implementation example by multiple QPs per interface
    and a single QP per protocol (except for one QP for IPv4,ARP,and RARP).]

1) At first, Master SM initializes ports in its IB-subnet and set them
   to ACTIVE.
2) In a node, a ipoib driver initializes itself.
  (create a QP for IPv4 broadcast MCG and IPv4,ARP,and RARP themselves
   ,join the broadcast MCG after getting p_key (which is assigned to each
    IP-subnet or may be shared by both IP on X subnet and IP on ipoib
subnet.)
   ,get Q_Key from the MCG
   ,if need,modify Q_Key of QP (or destroy and re-create a QP with a new
Q_Key)
   ,attach the QP to HCA's MCG table on the node.
   ,post receive-buffers to HCA)

3) When a "protocol"(X or Y) starts to use an ipoib interface,
  ipoib creates a QP for each protocol except for IPv4/ARP/RARP.
  If the protocol is IPv4 or ARP or RARP, it re-uses the IPv4 broadcast QP
  and does not create a new one.
  Every protocol do the same thing in its own way.

4) Here a QP for IPv4/ARP/RARP, a QP for X, and a QP for Y per ipoib
interface
   are ready to send and receive packets.

-- ARP on ipoib --
5) For example,
   ARP on ipoib in node A reqeusts for ipoib to send an ARP REQUEST packet
to
   the IPv4 broadcast MCG.

6-a)  Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. ARP packet)
    and passes this packet to ARP on ipoib.
   In this time, ipoib can get node A's GID, QPN-for-IPv4 in the packets,
    and LID in the result of poll-for-completion.
6-b) ARP on ipoib can get node A's IPv4 address on ipoib and MAC
     or invalidate node A's IPv4 address on ipoib IPaddr-MAC cache entry.

   Assumption:
      ipoib driver can distinguish protocol of the packet is ARP.

-- "ARP on X" --
7) ARP on X in node A reqeusts for X to send an ARP REQUEST packet to
   the IPv4 broadcast MCG,
   and X requests for ipoib to send a packet
   which encapsulates(encodes. e.g. add X's header) the ARP packet.

8-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. X packet) and
    and passes this packet to X on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
8-b) X on ipoib decodes(e.g. strips X's header) X packet into ARP packet,
    and passes this packet to ARP on X.
8-c) ARP on X can get node A's IPv4 address on X and MAC
     or invalidate node A's IPv4 address on X IPaddr-MAC cache entry.

   Assumption:
      ipoib driver can distinguish protocol of the packet is X by QPN.
      X can distinguish protocol of the packet is ARP. It is up to X.

-- "Y's own packet on ipoib" --
9) Y on ipoib in node A reqeusts for ipoib to send an Y's own packet to
   the IPv4 broadcast MCG.

10-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. Y packet) and
    and passes this packet to Y on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
10-b) Y on ipoib get node A's Y dependent info,

   Assumption:
      ipoib driver can distinguish protocol of the packet is Y.

Thanks,
Akihiko

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf
>Of [email-masked]
>Sent: Wednesday, February 27, 2002 10:45 PM
>To: [email-masked]
>Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>
>
>>Date: Wed, 27 Feb 2002 12:33:40 -0500 (EST)
>>From: Bill Strahm <[email-masked]>
>>Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>>
>>I think you misunderstand the layering ... First I would call your bottom
>>layer Infiniband.
>
>No, I wrote, in the figure, IPoIB as bottom. Of course, Infiniband (incl.
>HCA driver) is below IPoIB. I omit Infiniband suite below IPoIB in the
>figure.
>
>>  Then you have protocols x,y,IP running on top of IB.
>>From there I would claim that x is ARP, IP is IP and y is some protocol
>>foo (Lets say IPX for a concrete example).
>>
>>I would hope that you would put IPX on its own QP rather than try to
>>comingle it with IP traffic.  You could also put multiple "VLANs" on
>>seperate QP rather than on a single QP,
>
>Yes, it is possible. but putting Ethertype is simpler, I think.
>
>> but then ethertype won't help you
>>at all there,
>
>Why ? With ethertype, I don't have to use multiple QPs per interface
>and implementation for multiple protocols can be possible and simple.
>
>> or are you saying that each subnet will have to have a
>>"unique" ethertype to distinguish it.
>
>No. One protocol's ethertype is always unique, i.e. the same value is used
>in all subnets.
>
>When I defines multiple interfaces (a QP per interface) per HCA port,
>
>even if they exist in multiple different IP-subnets on one IB medium,
>all IPv4s on them have 0x800 as ethertype,
>,all Xs on IPoIB on them(i/fs) have the same ethertype. e.g. 0xeeee.
>,and all Ys on IPoIB on them(i/fs) have the same ethertype. e.g. 0xd000.
>(About X and Y, please see the figure below quoted in this mail.)
>
>Thanks,
>Akihiko
>
>>Bill
>>
>>On Wed, 27 Feb 2002 [email-masked] wrote:
>>
>>> >From: Vivek Kashyap <[email-masked]>
>>> >Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>>> >Date: Tue, 26 Feb 2002 10:58:17 -0800 (PST)
>>>
>>> >> If every protocol has its own Ethertype in a payload,
>>> >> an IPoIB driver can simply dispatch a received packet to a suitable
>protocol
>>> >>  stack or discard it (if no suitable stack does not exist) correctly.
>>> >
>>> >Yes, that is obvious and how things would get filtered. That was not
>>> >what I was saying. The point was that in IB the QP allows the
>>> >protocols to come pre-sorted i.e. you don't have to demux them again.
>>> >When using any of the non-IP headers you end up putting that function
>>> >back into the drivers.
>>>
>>> What you say is possible.
>>> But I disagree putting upper layer function into lower layer.
>>> Layered architecture should be kept, I think.
>>>
>>> Example.
>>> +----------+---+                      +---+--------+
>>> | IP       |   |                      |   |  IP    |
>>> +----+     | Y |                      | Y |    +---+
>>> | X  |     |   |                      |   |    | X |
>>> +----+-----+---+                      +---+----+---+
>>> |  ipoib       |                      |   ipoib    |
>>> +-+-----+--+-+-+                      +-+-+--+---+-+
>>>   |     |    |  (c)EtherType = 0xd000   |    |   |
>>>   |     |    +--------------------------+    |   |
>>>   |     |                                    |   |
>>>   |     |     (a)EtherType = 0x800           |   |
>>>   |     +------------------------------------+   |
>>>   |                                              |
>>>   |           (b)EtherType = 0xeeee (e.g.)       |
>>>   +----------------------------------------------+
>>>
>>>   IP,X,and Y all directly use IPoIB driver and have its own Ethertype
>each.
>>>     (0xd000, 0xeeee are just example values.)
>>>   X encapsulates IP packets.
>>>   Y does not encapsulate IP packets.
>>>
>>>   IP address of IP over X is different
>>>     from IP address of IP directly over ipoib driver.
>>>   They are in different IP subnets.
>>>
>>> >Since you are associating the IP address with this QP and resolving it
>>> >using IP processes there shouldn't be any question of non-IP
>>> >encapsulations. It seems, from your examples, that you want to
>>> >encapsulate IP in another set of headers. Then your secondary demux
>>> >module distributes the IP packets to IP interfaces.
>>>
>>> Yes. It is one of what I think. In abobe example, "X" demuxes and passes
>>>  packets to IP or processes them by itself.
>>>
>>> >Sure you can do that. You could do the same on another QP on the same
>>> >port too without loss of function. If used on the same QP wouldn't the
>>> >various end points supporting non-IP protocol X have to know who
>>> >supports this option?
>>>
>>> In this case, ipoib driver can pass only X's packets to "X" correctly,
>>>  and it it up to "X" if the packet should be just handled by "X" only or
>>>  should be passed to IP or not.
>>>
>>> > Thus we end up with IP interfaces that can't
>>> >talk to each other.
>>>
>>> In above figure,
>>>
>>> - IP directly over ipoib (so-called normal "IP on IB") can talk
>>>   with another IP directly over ipoib.
>>> - IP over X can talk with another IP over X.
>>> - IP over X can not talk directly with another IP directly over ipoib,
>>>   because they are in different IP subnets. (they share the same medium
>but
>>>   exist in different IP subnets and have different MTU.)
>>> - Y over ipoib can talk only with another Y over ipoib.
>>>
>>> >So was questioning if the non-IP protocols need to be considered in
>>> >IPoIB at all.
>>> >
>>> >Having said that, in practical terms when using a type in payload, an
>>> >end point will certainly be able to filter away what it doesn't want.
>>> >An administrator will have to ensure the 'right' type of interfaces
>>> >exist on the subnet.
>>>
>>> I don't think so.
>>> It is up to not administrator but IETF/ipoib driver
>>>  to decide to pass received packets to upper layers or discard them.
>>> And it is possible to do that.
>>>
>>> >> >An IP QPN is not a link like other media
>>> >>
>>> >> I think it is possible to say "IPoIB QPN is a link. (i.e. an
>interface)".
>>> >> # When I read IBTA documents, I thought "Ethernet(,FDDI,Tokenring) is
>>> >> # like a single QP medium", if follow IBTA way.)
>>> >
>>> >This is exactly what I meant by opening up the QP.
>>>
>>> Your model is clear, I also think.
>>> And I think "a single QP per interface with Ethertype in payload model"
>>>  is also clear.
>>>
>>> #Let me repeat partially what I said in my past mails.
>>>
>>> Your idea is very suitable to a single (Not layered) IPv4/IPv6 protocol
>>> itself. (Of course a single QP is also suitable.)
>>> But yours is not suitable to layered architecture, as I explained above,
>>> besides processing of QP for IPv4+ARP+RARP.
>>> Of course, it may be possible to implement by your model. But not
>suitable,
>>> I think. Ethertype in payload can solve it simply.
>>>
>>> What are merits of using multiple QPs ?
>>>
>>> Merits of using multiple QPs which I think are load-balancing and
>parallel
>>> processing, fail-over, and distributed processing, etc based by multiple
>QPs
>>> especially by using "multiple IP QPs" (not a single IP QP but multiple
IP
>QPs).
>>> They all can be done by using multiple interfaces (one interface has a
>>>  single QP.) well and simply.
>>>
>>> > The QP is briging
>>> >down the number of protocols to 2,
>>>
>>> > but by considering it as a link we
>>> >put the multiplexing problem back in.
>>>
>>> What is multiplexing problem ? Do you mean double dispach checks by
>>>  HCA and ipoib which you say ? Please explain more.
>>>
>>> >On IPv4 one could argue that the problem is there - since we have IP
>>> >and ARP - but it is limited to two easily distinguishable protocols.
>>> >On IPv6 QPN it doesn't exist. Therefore an implementation could just
>>> >take the received packet as it is and not worry about
>>> >types/headers/encapsulations. This model seems simpler.
>>> >
>>> >That was the original reason for suggesting different QPs for
>>> >different protocols.
>>>
>>> Sure, I can understand your model. But please read my comments above.
>>>
>>> Thanks,
>>> Akihiko
>>>
>>> >With 2^24 QPNs possible at a port it seemed that one could use
>>> >separate QPNs.
>>> >>
>>> >> Thanks,
>>> >> Akihiko
>>> >
>>> >--
>>> >Vivek Kashyap
>>> >Linux Technology Center, IBM
>>> >[email-masked]
>>> >[email-masked]
>>> >503 578 3422 (o)
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

今井です。
今井＠基開統です。
今井＠基開統５開発部です。

---------------------------------------------------------------------------
  今井昭彦  川崎市中原区上小田中4-1-1 川崎工場 本館
16階 北側
  TEL(内) 7113-8706,    TEL(外) 044-754-3456, 044-777-1113*38706

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib