Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

From: "Larsen, Roy K" <[email-masked]>
Date: Fri, 22 Feb 2002 13:30:38 -0800

Vivek,

I think you've done a good job outlining the possible approaches and their
ramifications.  I have only two minor nits.  It wasn't clear to me if you
were stating that methods (ii) and (iii) would require using separate QPs
for IPv4 and IPv6.  If so, I believe an implementation could choose to use a
single QP for all IPoIB protocols and remain completely interoperable with
all other implementations.  In which, case Jerry's observation is still
valid.

Second, I don't think there needs to be a concern about using a type field
in the face of arbitrary protocols.  Protocols can only send to a QP that
they have obtained the QPN and Q_Key for.  This would presume that one would
not advertise the QP/Q_KEY to protocols that the implementation is not
prepared to handle.  That said, I would agree that if the consensus is to
encapsulate type, using the existing Ethernet types makes sense.

Roy

-----Original Message-----
From: Vivek Kashyap [mailto:[email-masked]]
Sent: Friday, February 22, 2002 12:25 PM
To: [email-masked]
Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

> 
> One Comment about Vivek/Jerry's Feb.6 draft.
> 
> >> >The QPs advertised for IP
> >> >         communication MUST NOT be used for other protocols.
> >> 
> >> It is unclear if this can be enforced or worthwhile to restrict so why 
> >> state it.  If used for anything else, this is a local implementation 
> >> problem to resolve.
> >
> >        Yes, it was discussed and agreed (I think) that we will use
> >        SHOULD. 
> 
> I think this sentence should be deleted.
> Because I think it is not recommendation but implementation-dependent that
>  if QP(s) for IP is shared with others or not.
> 

This sentence is related to the comment in the draft about resolving
the encapsulation format. We should try and resolve it in the coming
10 days or so (March, 2nd is the cut-off for drafts).

I present the issues below and hopefully enough people will join in to
allow the chair to call a consensus.

MUST is equivalent to the IBA model: --- (i)

        The IB architecture identifies data streams based on the QP
        that is associated with them. It does not include a type
        field.  

        This reasoning would suggest (say) we use a unique QP for
        IPv6. This QP (IPv6_QP) would be the one resolved during ND
        and is therefore expected to send and receive only IPv6
        packets.

        Thus the protocols can be directly encapsulated in the IB
        payload. 

        This model therefore makes it a MUST to separate the data
        streams using the QPs.

The above model can be applied to all possible protocols on IB. Thus
we need to have a separate QP for IPv6 and a separate one for IPv4.
This is what was suggested in the draft with the caveat that the
section will be effected based on the resolution of the
'type'/encapsulation issue.

We get a separate QPN using ND for IPv6. Get one for IPv4 using ARP.
Now, we could choose to return two (one for IPv4 and one for ARP).

As far as I can tell the choice is to exchange only one QPN for IPv4
and related protocols (ARP and its op-codes).

Thus we are now called upon to distinguish between these two types of
packets. There are two ways to do this:

a) Protocol signature bits  --- (ii)

   The QP is filtering the packets to just the two protocols. The two
   protocol headers are sufficiently different to separate the
   protocols by considering the first bits in the protocols as
   defining a 'type' field.

   The driver thus checks the type to determine the upper layer but
   does not 'step over' the field when it passes the packet to the
   upper layer.

b) A 'type' in the payload --- (iii)

   One can add a type field in the payload that in turn indicates the
   encapsulated protocol. 

So, we can do one of (i), (ii) or (iii). All three will work. Of the
three, (ii) and (iii) are the better options if we want to keep the
QPN exchange (in the case of IPv4, IPv6 is unaffected) limited to only
one. One could define a separate encapsulation for IPv4 and v6 but I
doubt if many are keen on that.

That leaves the solution to (ii) or (iii). 

With (ii) we keep ourselves limited to the IP specific protocols -
such as IPv6/IPv4/ARP. This is really the scope of IPoIB and so is
fine. 

However, some people have voiced the opinion that they would really
like to use 'other' protocols over the same QP. This can be
accomplished by adding a 'type' in the payload.

This solution of course depends on the meaning of 'other'. If it is
arbitrary protocols then there is no good solution since the arbitrary
protocols may follow their own rules and hence clash with any
encapsulation scheme we choose.

We could alternatively say that the ethernet numbers for
IPv6/IPv4/ARP/RARP will be used for the respective encapsulation. We
don't restrict the use of other values - but the 'others' are 
restricted to ethernet numbers. An IPoIB interface driver, of course,
is free to reject anything other than IPv6/v4/ARP/RARP.

I am ok with either of (ii) or (iii). Depending on the consensus
(hopefully called in the next few days) the draft can be updated and
submitted by March 2nd.

Vivek
-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 22 Feb 2002 15:37:47 -0800 (PST)

> all other implementations.  In which, case Jerry's observation is still

[Which observation of Jerry ? Or, do you mean the comments in the mail
I appended (was from Akihiko).]

If we use (ii) or (iii) we can demultiplex the protocols on the same
QP. It is therefore completely upto the implementation to share or
not-share the QPs.

So, in either case we drop the recommendation in the draft.

> encapsulate type, using the existing Ethernet types makes sense.

Not really. If the desire to use 'other' extends to arbitrary
protocols, then those protocols need to agree on the format i.e. agree
to use the type. With limiting the 'other' protocols to ethertype we
are talking about protocols that agree to use the type field and also
agree on which type value belongs to whom.

May be I am confused but the following bothers me - it is an
interoperability issue:

On the whole, the 'other' protocols over IB is a weak argument since
we are then opening the IP QPN (received by IP address resolution) to
exchange non-IP protocols. 

Non-IP protocols sent to IP address ? or, proprietary headers for some
load-balancing/other solution sent to an IP address (and QPN)? How
would implementation X talk to Y which ends up encapsulating
differently or how do they decide to agree on the encapsulation? Do we
end up in the situation of specifying a SNAP/NOSNAP equivalent on
IPoIB interfaces? These other types should get a QPN of their own -
they are not connected to IP at all.

An IP QPN is not a link like other media - it is purely an IP only
link or at a maximum an IP/ARP link. So, such non-IP types really
don't help. This seems to suggest that we need to state that the types
are :IPv4,IPv6 and ARP/RARP.

But if we are limited to just these then why not just use the first
few distinguishing bits (which serve the same purpose) and avoid
handling a pseudo-header?

Vivek

---

From: [email-masked]
Date: Mon, 25 Feb 2002 22:26:22 +0900

>From: "Larsen, Roy K" <[email-masked]>
>Subject: RE: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt
>Date: Fri, 22 Feb 2002 13:30:38 -0800
>
>Vivek,
>

Agree. (I had the same question, but following mails solved it.)

>encapsulate type, using the existing Ethernet types makes sense.

Agree. I also think 'type' should be Ethertype defined by RFC 1700.

>Roy

Thanks,
Akihiko

---

From: [email-masked]
Date: Mon, 25 Feb 2002 22:44:34 +0900

>From: Vivek Kashyap <[email-masked]>
>Date: Fri, 22 Feb 2002 15:37:47 -0800 (PST)

>If the desire to use 'other' extends to arbitrary
>to use the type. 

Yes. I assume that. i.e. Every protocol over IPoIB must follow IETF/IPoIB
draft, I think.

>they are not connected to IP at all.

If every protocol has its own Ethertype in a payload, 
an IPoIB driver can simply dispatch a received packet to a suitable protocol
 stack or discard it (if no suitable stack does not exist) correctly. 

>An IP QPN is not a link like other media

I think it is possible to say "IPoIB QPN is a link. (i.e. an interface)".
# When I read IBTA documents, I thought "Ethernet(,FDDI,Tokenring) is
# like a single QP medium", if follow IBTA way.)

Thanks,
Akihiko

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 26 Feb 2002 10:58:17 -0800 (PST)

>  stack or discard it (if no suitable stack does not exist) correctly. 

Yes, that is obvious and how things would get filtered. That was not
what I was saying. The point was that in IB the QP allows the
protocols to come pre-sorted i.e. you don't have to demux them again.
When using any of the non-IP headers you end up putting that function
back into the drivers.

Since you are associating the IP address with this QP and resolving it
using IP processes there shouldn't be any question of non-IP
encapsulations. It seems, from your examples, that you want to
encapsulate IP in another set of headers. Then your secondary demux
module distributes the IP packets to IP interfaces.

Sure you can do that. You could do the same on another QP on the same
port too without loss of function. If used on the same QP wouldn't the
various end points supporting non-IP protocol X have to know who
supports this option? Thus we end up with IP interfaces that can't
talk to each other.

So was questioning if the non-IP protocols need to be considered in
IPoIB at all.

Having said that, in practical terms when using a type in payload, an
end point will certainly be able to filter away what it doesn't want.
An administrator will have to ensure the 'right' type of interfaces
exist on the subnet.

> # like a single QP medium", if follow IBTA way.)

This is exactly what I meant by opening up the QP. The QP is briging
down the number of protocols to 2, but by considering it as a link we
put the multiplexing problem back in.

On IPv4 one could argue that the problem is there - since we have IP
and ARP - but it is limited to two easily distinguishable protocols.
On IPv6 QPN it doesn't exist. Therefore an implementation could just
take the received packet as it is and not worry about
types/headers/encapsulations. This model seems simpler.

That was the original reason for suggesting different QPs for
different protocols.

With 2^24 QPNs possible at a port it seemed that one could use
separate QPNs.

---

From: [email-masked]
Date: Wed, 27 Feb 2002 20:55:03 +0900

>Date: Tue, 26 Feb 2002 10:58:17 -0800 (PST)

>When using any of the non-IP headers you end up putting that function

What you say is possible.
But I disagree putting upper layer function into lower layer.
Layered architecture should be kept, I think.

Example.
+----------+---+                      +---+--------+
| IP       |   |                      |   |  IP    |
+----+     | Y |                      | Y |    +---+
| X  |     |   |                      |   |    | X |
+----+-----+---+                      +---+----+---+
|  ipoib       |                      |   ipoib    |
+-+-----+--+-+-+                      +-+-+--+---+-+
  |     |    |  (c)EtherType = 0xd000   |    |   |
  |     |    +--------------------------+    |   |
  |     |                                    |   |
  |     |     (a)EtherType = 0x800           |   |
  |     +------------------------------------+   |
  |                                              |
  |           (b)EtherType = 0xeeee (e.g.)       |
  +----------------------------------------------+

  IP,X,and Y all directly use IPoIB driver and have its own Ethertype each.
    (0xd000, 0xeeee are just example values.)
  X encapsulates IP packets.
  Y does not encapsulate IP packets. 

  IP address of IP over X is different
    from IP address of IP directly over ipoib driver.
  They are in different IP subnets.

>module distributes the IP packets to IP interfaces.

Yes. It is one of what I think. In abobe example, "X" demuxes and passes
 packets to IP or processes them by itself.

>supports this option?

In this case, ipoib driver can pass only X's packets to "X" correctly,
 and it it up to "X" if the packet should be just handled by "X" only or
 should be passed to IP or not.

> Thus we end up with IP interfaces that can't
>talk to each other.

In above figure,

- IP directly over ipoib (so-called normal "IP on IB") can talk
  with another IP directly over ipoib.
- IP over X can talk with another IP over X.
- IP over X can not talk directly with another IP directly over ipoib,
  because they are in different IP subnets. (they share the same medium but
  exist in different IP subnets and have different MTU.)
- Y over ipoib can talk only with another Y over ipoib.

>An administrator will have to ensure the 'right' type of interfaces

I don't think so.
It is up to not administrator but IETF/ipoib driver
 to decide to pass received packets to upper layers or discard them.
And it is possible to do that.

>This is exactly what I meant by opening up the QP.

Your model is clear, I also think.
And I think "a single QP per interface with Ethertype in payload model"
 is also clear.

#Let me repeat partially what I said in my past mails.

Your idea is very suitable to a single (Not layered) IPv4/IPv6 protocol
itself. (Of course a single QP is also suitable.)
But yours is not suitable to layered architecture, as I explained above,
besides processing of QP for IPv4+ARP+RARP.
Of course, it may be possible to implement by your model. But not suitable,
I think. Ethertype in payload can solve it simply.

What are merits of using multiple QPs ?

Merits of using multiple QPs which I think are load-balancing and parallel
processing, fail-over, and distributed processing, etc based by multiple QPs
especially by using "multiple IP QPs" (not a single IP QP but multiple IP QPs).
They all can be done by using multiple interfaces (one interface has a
 single QP.) well and simply.

> The QP is briging
>down the number of protocols to 2,

> but by considering it as a link we

What is multiplexing problem ? Do you mean double dispach checks by
 HCA and ipoib which you say ? Please explain more.

>That was the original reason for suggesting different QPs for

Sure, I can understand your model. But please read my comments above.

Thanks,
Akihiko

---

From: Bill Strahm <[email-masked]>
Date: Wed, 27 Feb 2002 12:33:40 -0500 (EST)

I think you misunderstand the layering ... First I would call your bottom
layer Infiniband.  Then you have protocols x,y,IP running on top of IB.
From there I would claim that x is ARP, IP is IP and y is some protocol
foo (Lets say IPX for a concrete example).

I would hope that you would put IPX on its own QP rather than try to
comingle it with IP traffic.  You could also put multiple "VLANs" on
seperate QP rather than on a single QP, but then ethertype won't help you
at all there, or are you saying that each subnet will have to have a
"unique" ethertype to distinguish it.

Bill

---

From: [email-masked]
Date: Thu, 28 Feb 2002 13:44:48 +0900

>Date: Wed, 27 Feb 2002 12:33:40 -0500 (EST)
>From: Bill Strahm <[email-masked]>
>layer Infiniband.

No, I wrote, in the figure, IPoIB as bottom. Of course, Infiniband (incl. 
HCA driver) is below IPoIB. I omit Infiniband suite below IPoIB in the figure.
 
>  Then you have protocols x,y,IP running on top of IB.
>seperate QP rather than on a single QP,

Yes, it is possible. but putting Ethertype is simpler, I think.

> but then ethertype won't help you
>at all there,

Why ? With ethertype, I don't have to use multiple QPs per interface
and implementation for multiple protocols can be possible and simple.

> or are you saying that each subnet will have to have a

No. One protocol's ethertype is always unique, i.e. the same value is used 
in all subnets.

When I defines multiple interfaces (a QP per interface) per HCA port,

even if they exist in multiple different IP-subnets on one IB medium,
all IPv4s on them have 0x800 as ethertype,
,all Xs on IPoIB on them(i/fs) have the same ethertype. e.g. 0xeeee.
,and all Ys on IPoIB on them(i/fs) have the same ethertype. e.g. 0xd000.
(About X and Y, please see the figure below quoted in this mail.)

Thanks,
Akihiko

---

From: "Ashwin Kamath" <[email-masked]>
Date: Wed, 27 Feb 2002 22:59:30 -0600

Akihiko,
 How would you propose implementations working with multiple queuepairs for
different transports to interoperate with your implementation. Do you
propose that everybody use the ether-type and a single queue pair for all
ethernet traffic. Then we would be solving ethernet-over-IB and not ipoib,
is this right?

-Ashwin.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf
Of [email-masked]
Sent: Wednesday, February 27, 2002 10:45 PM
To: [email-masked]
Subject: Re: [Ipoverib] draft-ietf-ipoib-ip-over-infiniband-00.txt

No, I wrote, in the figure, IPoIB as bottom. Of course, Infiniband (incl.
HCA driver) is below IPoIB. I omit Infiniband suite below IPoIB in the
figure.

Thanks,
Akihiko

---

From: [email-masked]
Date: Thu, 28 Feb 2002 22:02:35 +0900

>From: "Ashwin Kamath" <[email-masked]>
>Date: Wed, 27 Feb 2002 22:59:30 -0600

Ashwin,

I try to write two examples (with/without Ethertype) using the following
 figure.

  node A                                        node B
+--------------+---+                      +---+--------------+
|IPv4/ARP/RARP |   |                      |   |IPv4/ARP/RARP |
+----+         | Y |                      | Y |          +---+
| X  |         |   |                      |   |          | X |
+----+---------+---+                      +---+----------+---+
|  ipoib           |                      |   ipoib          |
+-+-----+------+-+-+                      +-+-+--+-------+-+-+
  |     |        |  (c)EtherType = 0xd000   |    |         |
  |     |        +--------------------------+    |         |
  |     |                                        |         |
  |     |     (a)EtherType = 0x800/0x806/0x8035  |         |
  |     +----------------------------------------+         |
  |                                                        |
  |           (b)EtherType = 0xeeee (e.g.)                 |
  +--------------------------------------------------------+

>different transports to interoperate with your implementation.

> Do you
>ethernet traffic.

> Then we would be solving ethernet-over-IB and not ipoib,
>is this right?

>-Ashwin.

I don't understand why you say "ethernet traffic" and "ethernet-over-IB"
but I try to write.

# Sorry for only first broadcast packet send and receive processes examples.

-----------------------
["With Ethertype"
 An implemetation by a single QP per interface and multiple protocols per QP]

1) At first, Master SM initializes ports in its IB-subnet and set them
   to ACTIVE.
2) In a node, a ipoib driver initializes itself.
  (create only one QP for the ipoib interface
   ,join the broadcast MCG after getting p_key (which is assigned to each
    IP-subnet or may be shared by both IP on X subnet and IP on ipoib subnet.)
   ,get Q_Key from the MCG
   ,if need,modify Q_Key of QP (or destroy and re-create a QP with a new Q_Key)
   ,attach the QP to HCA's MCG table on the node.
   ,post receive-buffers to HCA)

3-a) When a "protocol"(IPv4,ARP,RARP,X or Y) starts to use an ipoib interface,
  ipoib gets Ethertype of the upper layer protocol.
3-b) When a "protocol"(IPv4,ARP,RARP) starts to use an X interface,
  X gets Ethertype of the upper layer protocol.

4) Here an only QP for every protocol (i.e. IPv4,ARP,RARP,X,Y) per ipoib
   interface is ready to send and receive packets.

-- "ARP on ipoib" --
5) For example,
   ARP on ipoib in node A reqeusts for ipoib to send an ARP REQUEST packet
   with Ethertype 0x806 on top of payload to the IPv4 broadcast MCG.

6-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. ARP packet) and
   sees that Ethertype in payload indicates ARP
    and passes this packet to ARP on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.

6-b) ARP on ipoib can get node A's IPv4 address on ipoib and MAC
     or invalidate node A's IPv4 address on ipoib IPaddr-MAC cache entry.

-- "ARP on X" --
7) ARP on X in node A reqeusts for X to send an ARP REQUEST packet to
   the IPv4 broadcast MCG,
   and X requests for ipoib to send a packet with Ethertype 0xeeee 
   which encapsulates(encodes. e.g. add X's header) the ARP packet.

8-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. X packet) and
   sees that Ethertype in payload indicates X
    and passes this packet to X on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
8-b) X on ipoib decodes(e.g. strips X's header) X packet into ARP packet,
    and passes this packet to ARP on X.
8-c) ARP on X can get node A's IPv4 address on X and MAC
     or invalidate node A's IPv4 address on X IPaddr-MAC cache entry.

   Assumption:
      X can distinguish protocol of the packet is ARP. It is up to X.

-- "Y's own packet on ipoib" --
9) Y on ipoib in node A reqeusts for ipoib to send an Y's own packet
   with Ethertype 0xd000 to the IPv4 broadcast MCG.

10-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. Y packet) and
   sees that Ethertype in payload indicates Y
    and passes this packet to Y on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
10-b) Y on ipoib get node A's Y dependent info,

---------------------
["Without Ethertype"
 An implementation example by multiple QPs per interface
    and a single QP per protocol (except for one QP for IPv4,ARP,and RARP).]

1) At first, Master SM initializes ports in its IB-subnet and set them
   to ACTIVE.
2) In a node, a ipoib driver initializes itself.
  (create a QP for IPv4 broadcast MCG and IPv4,ARP,and RARP themselves
   ,join the broadcast MCG after getting p_key (which is assigned to each
    IP-subnet or may be shared by both IP on X subnet and IP on ipoib subnet.)
   ,get Q_Key from the MCG
   ,if need,modify Q_Key of QP (or destroy and re-create a QP with a new Q_Key)
   ,attach the QP to HCA's MCG table on the node.
   ,post receive-buffers to HCA)

3) When a "protocol"(X or Y) starts to use an ipoib interface,
  ipoib creates a QP for each protocol except for IPv4/ARP/RARP.
  If the protocol is IPv4 or ARP or RARP, it re-uses the IPv4 broadcast QP
  and does not create a new one.
  Every protocol do the same thing in its own way.

4) Here a QP for IPv4/ARP/RARP, a QP for X, and a QP for Y per ipoib interface
   are ready to send and receive packets.

-- ARP on ipoib --
5) For example,
   ARP on ipoib in node A reqeusts for ipoib to send an ARP REQUEST packet to
   the IPv4 broadcast MCG.

6-a)  Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. ARP packet)
    and passes this packet to ARP on ipoib.
   In this time, ipoib can get node A's GID, QPN-for-IPv4 in the packets,
    and LID in the result of poll-for-completion.
6-b) ARP on ipoib can get node A's IPv4 address on ipoib and MAC
     or invalidate node A's IPv4 address on ipoib IPaddr-MAC cache entry.

   Assumption:
      ipoib driver can distinguish protocol of the packet is ARP.

-- "ARP on X" --
7) ARP on X in node A reqeusts for X to send an ARP REQUEST packet to
   the IPv4 broadcast MCG,
   and X requests for ipoib to send a packet
   which encapsulates(encodes. e.g. add X's header) the ARP packet.

8-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. X packet) and
    and passes this packet to X on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
8-b) X on ipoib decodes(e.g. strips X's header) X packet into ARP packet,
    and passes this packet to ARP on X.
8-c) ARP on X can get node A's IPv4 address on X and MAC
     or invalidate node A's IPv4 address on X IPaddr-MAC cache entry.

   Assumption:
      ipoib driver can distinguish protocol of the packet is X by QPN.
      X can distinguish protocol of the packet is ARP. It is up to X.

-- "Y's own packet on ipoib" --
9) Y on ipoib in node A reqeusts for ipoib to send an Y's own packet to
   the IPv4 broadcast MCG.

10-a) Every receive-ready node(node A and B in this case)'s ipoib
   in the IP-subnet receives the frame (incl. Y packet) and
    and passes this packet to Y on ipoib.
   In this time, ipoib can get node A's GID, QPN in the packets,
    and LID in the result of poll-for-completion.
10-b) Y on ipoib get node A's Y dependent info,

   Assumption:
      ipoib driver can distinguish protocol of the packet is Y.

Thanks,
Akihiko

>>> >With 2^24 QPNs possible at a port it seemed that one could use

今井です。
今井＠基開統です。
今井＠基開統５開発部です。

---------------------------------------------------------------------------
  今井昭彦  川崎市中原区上小田中4-1-1 川崎工場 本館 16階 北側
  TEL(内) 7113-8706,    TEL(外) 044-754-3456, 044-777-1113*38706