Subject: Re: [ipoverib] moving forward with the open Issues - end node

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 11 Feb 2002 15:29:47 -0800 (PST)

SG,

Having the "right" amount of info included in a standard track RFC is
always a challenge. Even between the two co-authors we feel differently -
I felt we can remove most of the MTU section, with a proposed addition
of the following paragraph to section 6.1 of the "link and multicast"
draft explaining "IP MTU":

> Note that in general, IP MTU - the payload size made available for the
> IP stack to use, can be set smaller than the full link MTU size. This
> flexibility makes it easier for one to bridge an IPoIB link with an
> Ethernet link by setting the IP MTU of all the connecting nodes to 1500
> bytes. For IPv4, this may require a manual configuration of an IP MTU
> different from the default, link MTU on all the nodes belonging to an
> IPoIB link. For IPv6, one can use the link MTU option of router
> advertisement [DISC] to announce a smaller IP MTU to all the nodes.

while Vivek felt the MTU section should stay. I do see some benefit of
inlcuding a short discussion on MTU issue for IB-Ethernet bridge like the
above as FYI only.

BTW, what is the PMTU need you alluded below? ("IP PMTU" or "IB PMTU"?)

Jerry

>Date: Sat, 09 Feb 2002 10:17:01 -0800
>From: "Sukanta Ganguly" <[email-masked]>
>To: "Vivek Kashyap" <[email-masked]>, [email-masked] (Michael Krause)
>Cc: [email-masked]
>Subject: Re: [ipoverib] moving forward with the open Issues - end node
>Mime-Version: 1.0
>Content-Transfer-Encoding: 8bit
>X-MIME-Autoconverted: from quoted-printable to 8bit by optimus.ietf.org id NAA28353
>X-Mailman-Version: 1.0
>List-Id: IP over InfiniBand WG Discussion List <ipoverib.ietf.org>
>X-BeenThere: [email-masked]
>
>Vivek,
>   Aren't we complicating the issues with MTU a bit too much. For IPoIB all that is needed 
is a mechanism to query and find out the MTU. What happens for IPv4 over Ethernet or IPv6 
over Ethernet is already in place and we do not need to define anything at that space. So I 
think we should leave the IPv4 over Ethernet and IPv6 over Ethernet MTU discovery and 
recommendations out of IPoIB. 
>   We need to have a IPoIB MTU query proposition in place which does not piggyback on any 
unreleted assumptions. On a similar note I do see a Path MTU needed for IPoIB, for e.g, IPoIB 
pdu traversing through multiple IB islands over an IP network.
>
>Does that make sense? Hope it does not raise too many questions? But please do shoot them to 
me if they do.
>
>SG
>
>*********** REPLY SEPARATOR  ***********
>
>On 2/8/2002 at 1:49 PM Vivek Kashyap wrote:
>
>>The IPoIB link MTU needs to be differentiated from the IP MTU
>>        on the IPoIB link. The IPoIB link MTU has to be decided based
>>        on the capacities of the underlying infrastructure comprising
>>        of the physical links, CAs, switches and IB routers. 
>>
>>	Ethernet LANs, which are very common, support an MTU of 1500
>>        bytes. The IPv6 specification further requires a minimum MTU
>>        of 1280 bytes. Therefore it is very appropriate to set the IP
>>        MTU to these values depending on the networking needs. It must
>>        however be ensured that the IPoIB link MTU is at least 2048
>>        bytes. IBA MTUs of smaller values are not optimal for
>>        internetworking to other IP subnets. Therefore it is
>>        RECOMMENDED that the IB infrastructure use components that can
>>        support an MTU of at least 2048 bytes. 
>>
>>        Since there are many considerations in deciding on the IP MTU,
>>        an implementation should choose the IP MTU according to the
>>        following priorities:
>>
>>	 IPv4:
>>
>>         1) The MTU associated with the broadcast GID associated with
>>            the IP subnet (IPoIB link MTU) but only if smaller than
>>            set by manual configuration.
>>
>>	 2) Manual configuration of the IP MTU 
>>
>>
>>	IPv6:
>>
>>         1) The MTU associated with the broadcast GID associated with
>>            the IP subnet (IPoIB link MTU). IPv6 nodes must however
>>            choose this value only if it is smaller than any router
>>            advertisements received.
>>
>>         2) Router advertisements, but only if not greater than manual
>>            configuration. 
>>
>>	 3) Manual configuration
>>
>>
>>        This way we have the situation that Mike/Yaron suggest - we
>>        recommend 2K of link MTU. An interface has multiple pieces of
>>        information to decide on the IP MTU. If the IPoIB node knows
>>        that there is a need to bridge across to other LANs it can
>>        drop down to the 1500 value for the IP MTU. In IPv6 nodes the
>>        IP MTU could drop down to 1280 bytes instead of 1500. One
>>        implicit limit is that the local port support the right MTU
>>        but that can be ignored since all the above happens after a
>>        successful join of the broadcast GID.
>>
>>        One aspect can be that we suggest a default of 1500 bytes for
>>        the IP MTU due to bridging requirements. However, if bridging
>>        is really a concern the bridge will not join in the IPoIB
>>        broadcast GID unless the physical MTU is the right size. 
>>
>>        By default I understand a value that is assigned automatically
>>        unless overridden. We always have a clear idea of what it has
>>        to be due to the IPoIB link MTU. So I don't think we need to
>>        suggest a default for the IP MTU. The same is true at the
>>        IPoIB link MTU level -the administrator considers the hardware
>>        available and sets the value. Hence a suggested or recommended
>>        value is enough.
>>
>>
>>Comments  ?
>>
>>Vivek
>>
>>> 
>>> At 04:12 PM 2/7/2002 -0800, Vivek Kashyap wrote:
>>> > >
>>> > > Yaron,
>>> > > Thanks for your response.
>>> > >
>>> > > BTW, I think "what IPoIB-default-MTU should be" is an open issue.
>>> > > right ?
>>> >
>>> >Akihiko,
>>> >
>>> >         We don't have a default MTU; the administrator is free to set
>>> >         any of the 5 possible values (256,512,1024,2048,4096 bytes)
>>> >         based on the capacity of the CAs/switches in the subnet.
>>> >
>>> >         In general using a minimum link MTU (and hence choosing the
>>> >         'right' equipment) of 2048 bytes makes sense. This link MTU
>>> >         allows for the IP MTU to be set to 1500 (ethernet
>>> >         internetworking) or 1280 (IPv6 minimum) if so desired. The
>>> >         smaller MTUs are not optimal.
>>> >
>>> >         No default though except in terms of :
>>> >
>>> >         If No-ethertype in payload
>>> >
>>> >                 IP MTU = IPoIB link MTU
>>> >
>>> >         else
>>> >                 IP MTU = IPoIB link MTU - sizeof('ethertype')
>>> 
>>> In general, one uses a 2KB or 4KB PMTU within an IB subnet and not a 
>>> smaller value (a smaller value does not meet the minimums needed for
>>IPv6 
>>> or an Ethernet I/O module as you noted).  When the PMTU is calculated
>>for 
>>> an endnode that flows through an IB I/O module, e.g. Ethernet, then the 
>>> PMTU is reduced accordingly.  I see no reason to not state that we have
>>a 
>>> default MTU of 2KB and then have the PMTU adjusted as required by the
>>path 
>>> this data flow follows.
>>> 
>>> Mike
>>> 
>>> 
>>> _______________________________________________
>>> IPoverIB mailing list
>>> [email-masked]
>>> https://www1.ietf.org/mailman/listinfo/ipoverib
>>> 
>>
>>
>>-- 
>>Vivek Kashyap
>>Linux Technology Center, IBM
>>[email-masked] 
>>[email-masked] 
>>503 578 3422 (o)
>>
>>_______________________________________________
>>IPoverIB mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Tue, 26 Feb 2002 10:54:32 -0800 (PST)

<snip>
>Majority of 1st gen is 2KB.  Majority of 2nd gen is 2-4KB.  2nd gen will be 
>available from various vendors fast enough that most implementations will 
>not suffer any schedule hits in getting this specification in place.
>
>>While I don't have any great vested interest in any existing hardware 
>>(beyond the 100 or so cards we use internally), I can't see why the use of 
>>a smaller MTU is so offensive. Sure, performance will be much better with 
>>a larger MTU....but, in my book, performance is what comes after
>>interoperability. Once you have everyone interoperating, performance 
>>enhancements and claims will naturally follow.
>
>Because most of us would rather support the existing IP RFCs that require 
>larger MTU support and most of us would rather avoid creating legacy 
>problems from an interoperability point of view.

Both drafts have clearly stated the 1280 minimal MTU size required by IPv6.
But for IPv4 the bar is much lower, thus allowing devices with smaller
MTU to join. The link and multicast draft also separates the definition
of IPv4 links from IPv6 links, making it possible to configure an
IPv4-only link (e.g. if one wants to connect an IB device with MTU of
1024).

The link and multicast draft also contains a "caveat" clause stating
once an IPoIB link MTU is chosen, devices of a smaller MTU won't be
able to join the IPoIB link unless the link MTU is reconfigured. I
hope that gives enough warning to the network adminstrator as far as
interoperability with smaller MTU devices is concerned.

Jerry

---

From: "Larsen, Roy K" <[email-masked]>
Date: Wed, 27 Feb 2002 09:11:04 -0800

As long as the IPoIB protocol does not require that an implementation must
refuse to join an all-nodes multicast group that has an MTU less than 2K,
this is all rather moot.  Is that what the draft authors are proposing?  If
not, the administrator of a fabric is free to set the MTU to whatever fits
the topology and IP protocols needed in that data center. 

Roy

> -----Original Message-----
> From: David Edmondson [mailto:[email-masked]]
> Sent: Wednesday, February 27, 2002 12:00 AM
> To: Bill Strahm
> Cc: Roger Ronald; Michael Krause; [email-masked]
> * [email-masked] [2002-02-26 19:08:45]
> > The problem that you run into is that while IPv4 has a very 
> small minimum
> > MTU requirement, IPv6 requires a much larger MTU.  ALL new protocols
> > coming out of the IETF have to be IPv6 aware, which means a 
> minimum MTU of
> > 1280 bytes ber the IPv6 RFCs.
> >
> > For this reason we have made the decision (and I believe 
> there is rough
> > concensus on this) to exclude devices that do not support at least a
> > minimum MTU of 2K.
> >
> > There is no reason why those devices can not do non-standard
> > encapsulations, or be configured to allow only IPv4 traffic 
> with a smaller
> > MTU, so I think this is the best of both worlds
> 
> In this last instance, with whom would those devices communicate ?  If
> there are other IPoIB devices which _require_ this device to support
> an MTU smaller than 2k, then they are not "IPoIB" devices.  If the
> other devices don't require the smaller MTU, why would one want to use
> it ?
>