Subject: Re: [Ipoverib] IPoIB architecture/encapsulation

From: "Vivek Kashyap" <[email-masked]>
Date: Mon, 10 Dec 2001 23:59:27 -0800

Jerry,
     I was asked to write an overview/architecture document.  It was called
requirements originally and I kept that in the draft file name (but not in
the title). I guess I should have resubmitted it as
...ipoib-architecture-00.txt.

See my comments below between <VK>

Vivek
--
Vivek Kashyap
[email-masked]
[email-masked]
Ph: 503 578 3422 T/L: 775 3422

                    "H.K. Jerry                                                                                   
                    Chu"                 To:     [email-masked]                                                
                    <Jerry.Chu@eng       cc:                                                                      
                    .sun.com>            Subject:     Re: [Ipoverib] IPoIB architecture/encapsulation             
                    Sent by:                                                                                      
                    ipoverib-admin                                                                                
                    @ietf.org                                                                                     

                    12/09/2001                                                                                    
                    11:26 PM                                                                                      
                    Please respond                                                                                
                    to "H.K. Jerry                                                                                
                    Chu"                                                                                          

Vivek,

I have a few comments on draft-kashyap-ipoib-requirements-01.txt.
I'm going to skip the IB overview and focus on IPoIB related sections.

First, what's the plan for this I-D? Is it intended to be informational,
or is it intended to get on the standard track?

I have also some serious concern, after reading past section 4, about
the intent of this i-d. It seems have gone way beyond its scope as a
"requirement" doc.

<VK>
     As noted above describes the IPoIB architecture.
<VK>

Section 3.2 Multicast support, I'd like to suggest making the IB
multicast support a requirement for running IP. Without a proper link
multicast support, it may signifcantly complicate the IPoIB standard.

This is not to say that implementation lacking hardware multicast
support can't emulate multicast function in software. But such
emulation must be made completely transparent to standard IPoIB stacks.

<VK> That is exactly what I say. IB multicast is mandatory. It is
recommended that one setup a fabric with multicast support for switches. If
the switches don't support it one will have to come up with some solution
at the IB level (which could be a multicast server). IPoIB wouldn't be
effected at all.  I will make this clearer in the draft. We are in
agreement here.
<VK>

Section 3.2.2 Transient flag in IB MGIDs, IBA, after failing to
includie P_Key as part of an IB multicast group definition, has
acknowledged the infeasibility of IB well known MGIDs. So there is not
much point to follow the rule that was set for IB well known MGIDs.

<VK> I am not sure of that, I think they said the well-known group for UD
mode doesn't make sense. The specification doesn't say it and it doesn't
remove raw IP mode's well known MGIDs. The IBTA is not killing Raw IP mode
(instead the management working group is talking about using that mode) and
the specification still says that the raw MGIDs are the well-known MGIDs.
Secondly IB's raw IPv6 mode doesn't recognise partitions. The SM doesn't
recognise partitions either. So in a subnet the raw mode and IPoIB mapped
MGIDs cannot exist if they share the MGID space. It is better to reduce
this conflict by using the transient bit.

     The only reason I am suggesting transient flags is to avoid such a
mixup if someone, and that someone would be compliant with IB spec, was
supporting the raw mode. Then there would be a clash with IPoIB MGIDs. One
of the modes wouldn't work. Since in this respect we are the upper layer we
might as well avoid such a clash. It buys us such avoidance and nothing
else.  We certainly don't loose anything.
<VK>

Section 3.4 Multicast address to LID mapping, IMO not much of
explanation is needed here because according to IB spec., LID is
allocated and managed by SM, period. (Clients can't specify a MLID when
creating a MCGroupRecord.)

<VK>
Yes, they can. See section 3.5.11.2.1.

 " 1) An (administrative) application defines (or determines) a target
multi-cast
   group address (GID). It specifies particular group characteristics
   (PMTU, raw, P-Key, etc.) and creates the multicast group by invoking
   a multicast group create to the multicast management entity. This
ap-plication
   may request a specific multicast LID or have one allocated
   for it. "

The discussion is included for the following reasons:

1) To clarify that in IPoIB the above option of specifying the LID is not
needed.

2) It has been suggested in this WG that there be a direct map from IP to
MLID. It was also discussed in the last IETF meet if I remember it
correctly. This discussion outlines the reasons for not providing such a
map.

<VK>

Subsection 1) Sending/receiving IP multicast, I don't think a
sender of a mulicast packet must be a member of that multicast group.
This is similar to IP over Ethernet. The only trouble for IB is that
there may not be a link mutlicast group to send this packet out. I have
suggested for such a case to forward the packet to a local multicast
router.

<VK>
It need not be a member at the IP level. It has to be at the IB level. It
may be that the switch connected to the sender is not part of the intended
multicast tree. In such a case the packet will not reach the intended
parties. The only way to ensure this is to join the IB multicast group and
be part of the tree.

If there is no MGID that is a different matter. You either create it or
send it out on the all-IP nodes/broadcast MGID. That is why the broadcast
MGID is a requirement in the IPoIB architecture. The router can certainly
pick it up from there. One cannot assume the MGID is not there because
there is no listener or that a router will do the job (since there may be
no router).

<VK>

4.0 IP subnets in InfiniBand fabrics, IMO it goes beyond the scope of
a requirement. and gets into some specific mechanics. Although what you
said there is what you and I have agreed upon, it's best to leave
this section to the multicast I-D we'll be working together.

<VK>
     The idea is to include all background information into the
architecture draft so that a reader gets the overall picture of IPoIB
setup. We don't have to repeat the same information in the
multicast/encapsulation drafts. Those drafts can focus on the actual
values, formats and mappings. Looks like we view the boundary in the
background information differently. Could it be because you got the
impression that this is a requirements draft  and I wrote it with the idea
of describing the IPoIB architecture ?

<VK>

4.1 IPoIB VLANs, again IMO you've deviated from the requirement and
got into specifics.

<VK>
The draft is only outlining what is needed for VLANs (architecture
document). It is not defining how things will be implemented. That is for
the draft on multicast.
<VK>

4.2 Multicast in IPoIB subnets, I'm not sure about the three levels
you described there. Which RFCs are these levels defined?

<VK>
     They are only for IPv4 as the draft notes. See RFC 1112 - Host
Extensions for IP multicasting.
<VK>

4.2.1 Sending IP multicast datagrams, I don't think what you
described is correct. An non-existing IB multicast group on the local
link only means that there is no member node of the group on the local
link. (I don't believe resources for MGID allocation will be a real
issue.) But a multicast router must attempt to forward the packet to
members in remote networks.

<VK>
     Creation of an MGID requires some change in the SM's state (the MGID
database and other things). It could fail due to any reason - a transient
error in the SM; a resource constraint; loss of packets; a random reason.
If such a thing happens the draft suggests that the packet be sent out
using the all-IP node/broadcast MLID/MGID.
This will cause all including the router to receive the multicast packet.
They can then filter it or forward it as needed. (Even if there was no
error and no MGID, and it is a pure sender, the sender will use the
broadcast MLID/MGID).

     The node in the meantime can retry to create/join the MGID based on
the error.
<VK>

4.2.2 Receiving multicast packets, it seems a bit redundant. There is
no need to mention IGMP/MLD or any specific protocols. Once IPoIB
multicast mapping is defined, things will fall into place automatically.

<VK>
     What is being described is the interaction at the IB level. The
sentence about IGMP/MLD was to explain that the report would not map to the
corresponding IP multicast address's MGID but the broadcast MGID/MLID. In
retrospect that explanation is not needed. I will remove it.
<VK>

After 4.2.2, I decided to stop because it becomes less and less clear
to me what the intent of this I-D is. You seem to have deviated
gradually from the 'requirement' spec., and included most of the
multicast material that you and I have agreed to work upon in a
separate draft. Can this draft be made to focus on "requirement" only?

<VK>
     The draft defines the IPoIB architecture however it doesn't go into
the actual mappings and encapsulations.
<VK>

Vivek

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/ipoverib

---

From: Vandana Rao <[email-masked]>
Date: Tue, 11 Dec 2001 08:31:52 -0800

At 11:59 PM 12/10/2001 -0800, Vivek Kashyap wrote:
<snip>

>    may request a specific multicast LID or have one allocated

Section 3.5.11.2 is incorrect and an erratum against this has already been 
filed by the IB MgtWG as it
is inconsistent with what is stated in the SA chapter (which is where the 
the compliance statements regarding multicast group create are defined). 
The MLID is always created by the SM/SA. The application has no interface 
available to specify MLIDs.

-Vandana

---

From: Michael Krause <[email-masked]>
Date: Tue, 11 Dec 2001 10:03:04 -0800

>this conflict by using the transient bit.

To be more precise, chapter 4 states a set of well-known IPv6 addresses 
that are considered reserved and notes additional RFC that should be 
consulted.  Further, raw IPv6 header usage indicates that this packet is a 
raw IPv6 packet and thus everything applying to IPv6 applies.  The problem 
with the raw mode is the lack of sufficiently strong CRC protection 
(applies to raw IPv6 and raw EtherType).  A number of people have 
determined that a 16-bit CRC is not sufficient which is the reason why we 
are all trying to use IB transports as a layer 2 construct in this effort, 
i.e. we can have strong data integrity at the IB transport layer.

>else.  We certainly don't loose anything.

IB MGIDs are not IPv6 addresses; IB reserves well-known IPv6 addresses to 
avoid this conflict from the start.

>Yes, they can. See section 3.5.11.2.1.

As noted in another e-mail, this is being changed via an errata.

>parties. The only way to ensure this is to join the IB multicast group and

But there may be many IB multicast groups and there may be many different 
IP to IB mappings within the same subnet(s).

>document). It is not defining how things will be implemented. That is for

Why are VLANs an issue at all?  VLANs are a layer 2 problem not an IP-to-IB 
problem.  For each VLAN, a separate IP-to-LAN driver instance is 
created.  For IB, VLANs are in essence different Q_Key or P_Key domains 
(though Q_Keys don't get filtered at switch level they can act as a filter 
if desired).  Thus, each IB VLAN is a separate IP-to-IB driver instance at 
the implementation level.  In general, I don't see how VLANs really matter 
that much per se as long the Q_Key and P_Key information is provided the 
implementation can decide how to set up their stacks accordingly.

>      The node in the meantime can retry to create/join the MGID based on

If there are no SM / fabric resources to establish the multicast group, 
then either the operation simply fails or one goes into promiscuous mode 
(just like Ethernet).  IB SM implementations will need to establish this 
all nodes multicast domain from the start (one per partition domain to be 
more exact) so that resources are guaranteed to be present.  After that, 
the SM can start to assign individual multicast groups their MGID / MLIDs 
from the IB perspective.

Mike