Subject: Re: [ipoverib] moving forward with the open Issues - end node

From: [email-masked]
Date: Fri, 01 Feb 2002 15:58:17 +0900

>Date: Fri, 01 Feb 2002 08:06:26 +0200
>From: Yaron Haviv <[email-masked]>
>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>
>It is from a different discussion, nothing to do with Ethertype field
>I suggested in the pass to enable a configuration of 1500MTU
>
>Yaron

Yaron, 
Thanks for your response.

BTW, I think "what IPoIB-default-MTU should be" is an open issue.
right ?

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>[email-masked]
>Sent: Friday, February 01, 2002 5:20 AM
>To: [email-masked]
>Cc: 'Michael Krause'; [email-masked]
>Subject: Re: [ipoverib] moving forward with the open Issues - end node
>
>>Date: Thu, 31 Jan 2002 20:36:34 +0200
>>From: Yaron Haviv <[email-masked]>
>>Subject: RE: [ipoverib] moving forward with the open Issues - end node
>>Mike,
>>
>>If you look at my last email from yesterday
>>I acknowledge that, and say that we need to look into in the future and not
>>in this RFC, so this issue isn't open anymore
>>The only open issue currently is the Ethertype field
>>
>>Yaron
>
>Please let me confirm this default MTU issue.
>
>If Ethertype is not in payload, default IP-MTU is 2048.
>Otherwise (=if Ethertype is in payload),
> default IP-MTU is not 2048.
>It's 2048 - EterhType field length. (4? probably not 2)
>
>>-----Original Message-----
>>From: [email-masked] [mailto:[email-masked]]On Behalf Of
>>Michael Krause
>>Sent: Thursday, January 31, 2002 6:23 PM
>>To: [email-masked]
>>Subject: Re: [ipoverib] moving forward with the open Issues - end node
>>
>>At 10:40 PM 1/29/2002 -0800, Vivek Kashyap wrote:
>>>Yaron,
>>>         The MCGroupRecord provides the IPoIB link MTU not IP MTU. The
>>>         IP MTU is set at the interface - by default implmentations
>>>         would set it to the link MTU itself.
>>
>>Assumption is that the multicast PMTU while at least on one or more IB
>>subnets is set to 2KB or 4KB (function of what the underlying fabric
>>supports).  For unicast, the normal IP PMTU discovery process is used and
>>nothing new is required to be specified.
>>
>>Mike
>
>Thanks,
>----------------------------------------------------------------------------
>--
>Akihiko Imai/Development Dept.5, Infrastructure Systems Division, Fujitsu
>Ltd.
>  e-mail: [email-masked]  TEL: +81-044-754-3456
>(ext)7113-8706
>  address: 4-4-1 Kamiodanaka Nakahara-ku, Kawasaki 211-8588, Japan
>
>_______________________________________________
>IPoverIB mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/ipoverib
>

---------------------------------------------------------------------------
  今井昭彦  川崎市中原区上小田中4-1-1 川崎工場 本館 16階 北側
  TEL(内) 7113-8706,    TEL(外) 044-754-3456, 044-777-1113*38706

_______________________________________________
IPoverIB mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipoverib

---

From: Vivek Kashyap <[email-masked]>
Date: Thu, 7 Feb 2002 16:12:06 -0800 (PST)

> BTW, I think "what IPoIB-default-MTU should be" is an open issue.

Akihiko,

        We don't have a default MTU; the administrator is free to set
        any of the 5 possible values (256,512,1024,2048,4096 bytes)
        based on the capacity of the CAs/switches in the subnet.

        In general using a minimum link MTU (and hence choosing the
        'right' equipment) of 2048 bytes makes sense. This link MTU
        allows for the IP MTU to be set to 1500 (ethernet
        internetworking) or 1280 (IPv6 minimum) if so desired. The
        smaller MTUs are not optimal.

        No default though except in terms of :

        If No-ethertype in payload

		IP MTU = IPoIB link MTU 

	else
		IP MTU = IPoIB link MTU - sizeof('ethertype')

Vivek

---

From: [email-masked]
Date: Fri, 08 Feb 2002 10:20:34 +0900

Vivek,

>		IP MTU = IPoIB link MTU - sizeof('ethertype')

It cleared ambiguity to me. Thanks.

--
Akihiko

---

From: Michael Krause <[email-masked]>
Date: Fri, 08 Feb 2002 06:56:09 -0800

>                 IP MTU = IPoIB link MTU - sizeof('ethertype')

In general, one uses a 2KB or 4KB PMTU within an IB subnet and not a 
smaller value (a smaller value does not meet the minimums needed for IPv6 
or an Ethernet I/O module as you noted).  When the PMTU is calculated for 
an endnode that flows through an IB I/O module, e.g. Ethernet, then the 
PMTU is reduced accordingly.  I see no reason to not state that we have a 
default MTU of 2KB and then have the PMTU adjusted as required by the path 
this data flow follows.

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Fri, 8 Feb 2002 13:49:19 -0800 (PST)

The IPoIB link MTU needs to be differentiated from the IP MTU
        on the IPoIB link. The IPoIB link MTU has to be decided based
        on the capacities of the underlying infrastructure comprising
        of the physical links, CAs, switches and IB routers. 

	Ethernet LANs, which are very common, support an MTU of 1500
        bytes. The IPv6 specification further requires a minimum MTU
        of 1280 bytes. Therefore it is very appropriate to set the IP
        MTU to these values depending on the networking needs. It must
        however be ensured that the IPoIB link MTU is at least 2048
        bytes. IBA MTUs of smaller values are not optimal for
        internetworking to other IP subnets. Therefore it is
        RECOMMENDED that the IB infrastructure use components that can
        support an MTU of at least 2048 bytes. 

        Since there are many considerations in deciding on the IP MTU,
        an implementation should choose the IP MTU according to the
        following priorities:

	 IPv4:

         1) The MTU associated with the broadcast GID associated with
            the IP subnet (IPoIB link MTU) but only if smaller than
            set by manual configuration.

	 2) Manual configuration of the IP MTU 

	IPv6:

         1) The MTU associated with the broadcast GID associated with
            the IP subnet (IPoIB link MTU). IPv6 nodes must however
            choose this value only if it is smaller than any router
            advertisements received.

         2) Router advertisements, but only if not greater than manual
            configuration. 

	 3) Manual configuration

        This way we have the situation that Mike/Yaron suggest - we
        recommend 2K of link MTU. An interface has multiple pieces of
        information to decide on the IP MTU. If the IPoIB node knows
        that there is a need to bridge across to other LANs it can
        drop down to the 1500 value for the IP MTU. In IPv6 nodes the
        IP MTU could drop down to 1280 bytes instead of 1500. One
        implicit limit is that the local port support the right MTU
        but that can be ignored since all the above happens after a
        successful join of the broadcast GID.

        One aspect can be that we suggest a default of 1500 bytes for
        the IP MTU due to bridging requirements. However, if bridging
        is really a concern the bridge will not join in the IPoIB
        broadcast GID unless the physical MTU is the right size. 

        By default I understand a value that is assigned automatically
        unless overridden. We always have a clear idea of what it has
        to be due to the IPoIB link MTU. So I don't think we need to
        suggest a default for the IP MTU. The same is true at the
        IPoIB link MTU level -the administrator considers the hardware
        available and sets the value. Hence a suggested or recommended
        value is enough.

Comments  ?

Vivek

-- 
Vivek Kashyap
Linux Technology Center, IBM
[email-masked] 
[email-masked] 
503 578 3422 (o)

---

From: "Sukanta Ganguly" <[email-masked]>
Date: Sat, 09 Feb 2002 10:17:01 -0800

Vivek,
   Aren't we complicating the issues with MTU a bit too much. For IPoIB all that is needed is a mechanism to query and find out the MTU. What happens for IPv4 over Ethernet or IPv6 over Ethernet is already in place and we do not need to define anything at that space. So I think we should leave the IPv4 over Ethernet and IPv6 over Ethernet MTU discovery and recommendations out of IPoIB. 
   We need to have a IPoIB MTU query proposition in place which does not piggyback on any unreleted assumptions. On a similar note I do see a Path MTU needed for IPoIB, for e.g, IPoIB pdu traversing through multiple IB islands over an IP network.

Does that make sense? Hope it does not raise too many questions? But please do shoot them to me if they do.

SG

*********** REPLY SEPARATOR  ***********

---

From: Michael Krause <[email-masked]>
Date: Mon, 11 Feb 2002 07:50:04 -0800

At 10:17 AM 2/9/2002 -0800, Sukanta Ganguly wrote:
>Vivek,
>    Aren't we complicating the issues with MTU a bit too much. For IPoIB 
> all that is needed is a mechanism to query and find out the MTU. What 
> happens for IPv4 over Ethernet or IPv6 over Ethernet is already in place 
> and we do not need to define anything at that space. So I think we should 
> leave the IPv4 over Ethernet and IPv6 over Ethernet MTU discovery and 
> recommendations out of IPoIB.
>    We need to have a IPoIB MTU query proposition in place which does not 
> piggyback on any unreleted assumptions. On a similar note I do see a Path 
> MTU needed for IPoIB, for e.g, IPoIB pdu traversing through multiple IB 
> islands over an IP network.

I believe all that is being stated by Vivek and myself is that to support 
IPv6 one needs to have a minimum MTU which for IB would require at least a 
2KB MTU be implemented within the fabric.

For an I/O controller attached via an IB TCA, e.g. Ethernet, this 
communication is independent of the IP over IB requirements.  Typically 
this is implemented using the IB RC or UC transport services and thus can 
operate using any size of IB MTU.  For an IB / IP router, one may use 
either RC or UC and operate using any of the IB MTU or one may use IB UD 
and simply act as a forwarding engine.  In both of these communication 
paradigms, PMTU services would need to be invoked at the IP level so that 
IP fragmentation is minimized where appropriate (e.g. for TCP-based 
communications).

Mike

---

From: Vivek Kashyap <[email-masked]>
Date: Tue, 12 Feb 2002 00:35:55 -0800 (PST)

> 
> Vivek,
>    Aren't we complicating the issues with MTU a bit too much. For IPoIB all that is needed is a mechanism to query and find out the MTU.

        No. It is very pertinent to explain why a particular value may
        be prefered. Things are made straightforward else we wouldn't
        be having this thread.

        The very reason people want a minimum of 2048 is due to
        ethernet and the 1280 limit of IPv6. In earlier mails in this
        thread a default of 2K or 1.5K has been proposed precisely for
        interoperating with IPv6/ethernet. So a recommendation to use
        a minimum of 2K and the reason for it are very relevant.

        The IP MTU is selected based on the criteria listed below in
        my earlier mail- the priorities clarify the process of
        selecting an IP MTU. Any of the various IP overFoo documents
        define these steps. They become even more necessary when there
        is no single default value to choose from.

What happens for IPv4 over Ethernet or IPv6 over Ethernet is already in place and we do not need to define anything at that space. So I think we should leave the IPv4 over Ethernet and IPv6 over Ethernet MTU discovery and recommendations out of IPoIB. 
>    We need to have a IPoIB MTU query proposition in place which does not piggyback on any unreleted assumptions. On a similar note I do see a Path MTU needed for IPoIB, for e.g, IPoIB pdu traversin…

        I am not at all clear as to what you mean. If you mean the
        IPoIB link MTU; it is determined by the administrator when the
        all-IPv6 nodes/broadcast group MGIDs are setup. IF the IP
        subnet spans multple IB subnets then the MGID will span the
        multiple IB subnets (a larger scope will be used). The IPoIB
        link will still be setup the same way.

Vivek