Subject: "RFC 2461bis" issue: MTU handling

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Oct 2003 09:09:26 +0200

I'm not sure this should go into a replacement specification for RFC 
2461, but I'll bring it up anyway:

Currently, routers can advertise an MTU for a link. That's nice. But 
what we really need is a way for hosts to find out the MTU each 
individual neighbor can handle. 100 Mbps and slower ethernet interfaces 
can typically handle only the standard 1500 byte ethernet MTU, while 
gigabit ethernet interfaces usually support a much larger MTU.

However, in most cases hosts with different MTUs are present on the 
same subnet, so simply advertising a larger MTU wouldn't solve this. 
(Not that this would work anyway as hosts are instructed to only listen 
to MTU advertisements where the MTU is between 1280 and 1500 (for 
ethernet).)

But if hosts can tell each other the MTU they support, each set of two 
hosts is always able to use the largest possible MTU between them. 
(This would also require a new link MTU option that conveys the maximum 
MTU the lower layer equipment supports. Switches have their own MTU and 
even some hubs start doing strange things when a larger than expected 
MTU is used.)

BTW, some duplication seems to have crept into the document:

    variable MTU   - a link that does not have a well-defined MTU (e.g.,
                     IEEE 802.5 token rings).  Many links (e.g.,
                     Ethernet) have a standard MTU defined by the link-
                     layer protocol or by the specific document
                     describing how to run IP over the link layer.

      variable MTU   - Neighbor Discovery allows routers to specify a MTU
                       for the link, which all nodes then use.  All nodes
                       on a link must use the same MTU (or Maximum
                       Receive Unit) in order for multicast to work
                       properly.  Otherwise when multicasting a sender,
                       which can not know which nodes will receive the
                       packet, could not determine a minimum packet size
                       all receivers can process.

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: [email-masked] (Jun-ichiro itojun Hagino)
Date: Thu, 23 Oct 2003 16:13:11 +0900 (JST)

> even some hubs start doing strange things when a larger than expected 

	the assumption made in RFC2461 is that the link MTU is constant
	over the link, i guess.  i don't think it is necessary to make
	MTU negotiable between peers, it would complicate too many things.

itojun

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Oct 2003 09:23:56 +0200

> 	MTU negotiable between peers, it would complicate too many things.

What would it complicate?

Obviously nobody is going to force anyone to implement such an option; 
it would only be used between consenting hosts.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 23 Oct 2003 10:49:36 +0300 (EEST)

On Thu, 23 Oct 2003, Jun-ichiro itojun Hagino wrote:
[...]
> 	MTU negotiable between peers, it would complicate too many things.

FWIW, I agree with this assumption.  Really, if you want to put different
MTU's on the same link and want to optimize it, use multiple links..

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Oct 2003 10:26:18 +0200

> MTU's on the same link and want to optimize it, use multiple links..

Could you people be any more dismissive if you tried?

I wish I could sentence you both to three months NOC duty.

See http://sd.wareonearth.com/~phil/net/jumbo/ for some links 
surrounding this issue.

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 23 Oct 2003 11:08:32 +0200 (CEST)

> even some hubs start doing strange things when a larger than expected 

This might be useful when the L2 doesn't have any MTU limitations.
For instance, with IP over ATM the default MTU is 9k but AAL5 has a 16 bit 
length field (if I don't misremember). Thus a pair of nodes can agree to use
e.g. 37.5k MTU.

But for links where the L2 has tigher limits things are quite a bit more
difficuly. Taking Ethernet as an example. Having two hosts say that they 
want to use a 9k MTU over Ethernet is fine as long as the bridges/switches
on the path all support that. But how can the hosts know that?
Even if the hosts can determine this (send a 9k packet and see if it gets
through?), what happens when an addition Ethernet switch or wire comes up
(or goes away) and spanning tree changes the L2 path between those
two hosts so that the packets no longer go through the same set of
bridges/switches?

If one would want to pursue this I think the best way would be for
IEEE 802 to define a "frame size discovery" protocol at L2 so that
hosts can robustly determine the frame size a given destination
MAC address and be notified when it changes.
But since Ethernet jumboframes are non-standard (even though more and more
commonly used it seems) and there is concern about the coverage of the Ethernet
CRC for larger frames, the IEEE has been unvilling to look into this.

Once you have such L2 capability then a ND option for "I can receive packets
with this MTU" allowing the a larger value than the default link MTU,
would make sense to me. But without the L2 capability it doesn't seem
very useful.

   Erik

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Oct 2003 13:11:50 +0200

On 23 okt 2003, at 11:08, Erik Nordmark wrote:

[Having hosts discover and use a larger than standard MTU between them]

> For instance, with IP over ATM the default MTU is 9k but AAL5 has a 16 
> bit length field (if I don't misremember). Thus a pair of nodes can 
> agree to use e.g. 37.5k MTU.

Isn't the MTU for IP over ATM something around 4500 bytes? And ATM is 
mostly a point-to-point protocol where hosts with a larger MTU are 
restricted by the need to be compatible with hosts with a regular MTU 
isn't relevant.

> But for links where the L2 has tigher limits things are quite a bit 
> more difficuly. Taking Ethernet as an example. Having two hosts say 
> that they want to use a 9k MTU over Ethernet is fine as long as the 
> bridges/switches on the path all support that.

Yes, this is an important limitation.

> But how can the hosts know that?

The simple solution would be for routers to announce the maximum packet 
size to be used on a subnet.

> Even if the hosts can determine this (send a 9k packet and see if it 
> gets through?), what happens when an addition Ethernet switch or wire 
> comes up (or goes away) and spanning tree changes the L2 path between 
> those two hosts so that the packets no longer go through the same set 
> of bridges/switches?

In theory it should be possible to create some kind of layer 2 PMTUD. 
But in practice I would expect operators to simply limit the MTU for 
the full subnet to the minimum of the MTUs supported by all switches on 
the subnet.

> MAC address and be notified when it changes.

Such a capability would enhance a layer 3 neighbor MTU management 
capability, but it isn't required. Operators can simply have routers 
annouce an upper limit on the MTU that may be used on a subnet. This 
doesn't impede the best case scenario where all the switches support a 
good sized MTU. Since the number of switches per subnet is typically 
much smaller than the number of hosts per subnet, getting all switches 
to support jumboframes is significantly simpler than getting all hosts 
to do the same.

> But since Ethernet jumboframes are non-standard (even though more and 
> more commonly used it seems) and there is concern about the coverage 
> of the Ethernet CRC for larger frames, the IEEE has been unvilling to 
> look into this.

Well we use a non-(IEEE-)standard way to encapsulate IP in ethernet 
frames (Ethernet II rather than 802.3) so we're not without precedent 
here.

I'm not a math or physics guy, but I'm not necessarily convinced by the 
CRC arguments. The ethernet CRC is pretty strong, it's not going to 
immediately break down at 9000 bytes or so, especially not in wired 
networks with very low bit error rates. And if the IEEE picks this up 
there is no reason they can't drop in a stronger CRC. As for the 
TCP/UDP checksum... yes, this is a problem (even at 1500 or ~4500 bytes 
that many IP over xxx RFCs specify) but why would the IEEE care?

> Once you have such L2 capability then a ND option for "I can receive 
> packets with this MTU" allowing the a larger value than the default 
> link MTU, would make sense to me. But without the L2 capability it 
> doesn't seem very useful.

But what would be the usefulness of such a layer 2 feature if higher 
layers don't take advantage of it? I think it's the other way around: 
having the layer 3 feature is useful in itself, when we have this the 
layer 2 feature is a nice bonus.

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 23 Oct 2003 14:09:44 +0200 (CEST)

> to support jumboframes is significantly simpler than getting all hosts 

Ah - now I get where you are going with this. From my "host perspective"
it is the switches lack of support which is the main problem - but you
are right that the inverse problem also exists.

So one approach would be something like having a router advertisement 
option that asserts "trust us; the L2 can in fact support 9k" resulting
in individual hosts deciding to send out an option with their NS/NA saying 
"I can support receiving 9k".

Note that this is different than the default MTU being 9k - since not all
hosts support the larger MTU everybody would still need to use the default
1500 MTU for sending broadcast and multicast packets on the link.

This still has the operational issue of what happens when
I need extra ports in my office and I get a cheap 4-port hub; neither
the IT department nor my hosts knows that this box is present and it
might not support jumboframes. One way to cope with this particular topology,
but no other topologies of varying frame size switches, would be
for the host to exchange a 9k packet with the router before
deciding that it should declare itself 9k capable.

  Erik

---

From: Fred Templin <[email-masked]>
Date: Thu, 23 Oct 2003 10:01:11 -0700

Hello Iljitsch,

An example use case is that on certain multiple-access wireless links
the optimal MTU for a particular neighbor may be proportional to
certain QoS metrics, e.g.,  the signal to noise ratio sensed at the
receiver's MAC layer.

I submitted a draft proposal back in January 2003 (attached below)
that suggests one possible approach by allowing NA messages to
include an MTU option (should I submit an update?).

Another approach would be to allow different semantics for the
interpretation of MTU options in RA messages based on the link type.

Comments?

Fred
[email-masked]

Iljitsch van Beijnum wrote:

> individual neighbor can handle. 100 Mbps and slower ethernet 
> interfaces can typically handle only the standard 1500 byte ethernet 
> MTU, while gigabit ethernet interfaces usually support a much larger MTU.
>
> (Not that this would work anyway as hosts are instructed to only 
> listen to MTU advertisements where the MTU is between 1280 and 1500 
> (for ethernet).)
>
> (This would also require a new link MTU option that conveys the 
> maximum MTU the lower layer equipment supports. Switches have their 
> own MTU and even some hubs start doing strange things when a larger 
> than expected MTU is used.)
>

Network Working Group                                         F. Templin
Internet-Draft                                                     Nokia
Expires: August 1, 2003                                 January 31, 2003

                 MTU Issues in IPv6 Neighbor Discovery
                   draft-ietf-templin-ndiscmtu-00.txt

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at http://
   www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on August 1, 2003.

Copyright Notice

   Copyright (C) The Internet Society (2003).  All Rights Reserved.

Abstract

   This document discusses Maximum Transmission Unit (MTU) issues in
   IPv6 Neighbor Discovery and suggests minor augmentations to the
   existing specification to rectify the issues.

1. Introduction

   This document discusses Maximum Transmission Unit (MTU) issues in
   Neighbor Discovery for IP Version 6 (IPv6) [1].  It argues that the
   current specification is too restrictive in the use of MTU options,
   and that per-neighbor MTU values should be maintained in the
   conceptual Neighbor Cache.  It finally proposes minor augmentations
   to the existing specification to rectify the issues.

Templin                  Expires August 1, 2003                 [Page 1]

Internet-Draft    MTU Issues in IPv6 Neighbor Discovery     January 2003

2. Problem Statement

   ([1], section 4.6.4) defines a Maximum Transmission Unit (MTU) option
   type.  In the current specification, the MTU option is sent only in
   Router Advertisement messages ([1], section 4.2) and interpreted by
   receivers as the MTU value for all nodes on the link to use.  While
   this specification provides sufficient mechanism for many of the
   supported link types in ([1], section 2.2), it may lead to
   inefficiencies for other types, e.g., for certain variable MTU links.

   In particular, it may be desirable in some cases for a node to track
   independent MTU values for different neighbors on a link.  For
   example, on certain multiple-access wireless links the optimal MTU
   for a particular neighbor may be proportional to the signal to noise
   ratio sensed at the receiver's MAC layer.  In another example,
   constrained nodes on a link with large MTU may wish to receive
   smaller packets than more robust nodes.  In these and other cases,
   maintaining independent per-neighbor MTUs for nodes on a link would
   yield significant efficiency advantages.

3. Proposed Changes

   In order to support per-neighbor MTUs, the following changes/
   augmentations to RFC 2461 are proposed:

   1.  In ([1], section 4.6.4), add new text allowing the encoding of
       MTU options in Neighbor Acknowledgement messages.

   2.  In ([1], section 5.1), add a per-neighbor MTU field (NBR_MTU) in
       the Neighbor Cache data structure.

   3.  In ([1], section 6.3.2), add a new host variable, defined as
       follows:

       MaxLinkMTU
          The maximum MTU supported on the link.
          Default: LinkMTU.

   4.  In ([1], section 7.2.4), add new text allowing the encoding of
       MTU options in solicited Neighbor Advertisements."

   5.  In ([1], section 7.2.5), add new text requiring the value in MTU
       options received in Neighbor Advertisement messages be written
       into the NBR_MTU field in the Neighbor Cache entry.

   6.  In (([1], section 7.2.6), add new text allowing Unsolicited
       Neighbor Advertisements sent to the unicast address of a
       neighbor.

Templin                  Expires August 1, 2003                 [Page 2]

Internet-Draft    MTU Issues in IPv6 Neighbor Discovery     January 2003

4. Operational Details

   When nodes implement the changes proposed above, Neighbor
   Advertisement messages containing MTU options provide a dynamic
   mechanism for receivers to inform senders of MTU changes.  Documents
   that specify the operation of IPv6 over specific link layers (e.g.,
   Ethernet, FDDI, etc.) provide details for the encoding of MTU options
   in Neighbor Advertisement messages.

   Receipt of an MTU option in the initial solicited Neighbor
   Advertisement provides an indication to the sender that the receiver
   implements the dynamic MTU mechanism.  (Lack thereof conversely
   provides an indication that the receiver does not implement the
   dynamic MTU mechanism.)

   Using these proposed changes, MTU options received in Router
   Advertisements affect LinkMTU exactly as in the current
   specification.  However, packetization and forwarding layers see an
   MTU of MaxLinkMTU when they examine the link.  The relationship
   between the three MTU parameters is as follows:

      LinkMTU <= NbrMTU <= MaxLinkMTU

   Multicast destinations, and unicast destinations that use a next hop
   with no MTU support indicated in the Neighbor Cache, see an MTU of
   LinkMTU.

5. IANA Considerations

   TBD

6. Security considerations

   TBD

7. Acknowledgements

   TBD

Normative References

   [1]  Narten, T., Nordmark, E. and W. Simpson, "Neighbor Discovery for
        IP Version 6 (IPv6)", RFC 2461, December 1998.

Informative References

Templin                  Expires August 1, 2003                 [Page 3]

Internet-Draft    MTU Issues in IPv6 Neighbor Discovery     January 2003

Author's Address

   Fred L. Templin
   Nokia
   313 Fairchild Drive
   Mountain View, CA  94110
   US

   Phone: +1 650 625 2331
   EMail: [email-masked]

Templin                  Expires August 1, 2003                 [Page 4]

Internet-Draft    MTU Issues in IPv6 Neighbor Discovery     January 2003

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   intellectual property or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; neither does it represent that it
   has made any effort to identify any such rights.  Information on the
   IETF's procedures with respect to rights in standards-track and
   standards-related documentation can be found in BCP-11.  Copies of
   claims of rights made available for publication and any assurances of
   licenses to be made available, or the result of an attempt made to
   obtain a general license or permission for the use of such
   proprietary rights by implementors or users of this specification can
   be obtained from the IETF Secretariat.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights which may cover technology that may be required to practice
   this standard.  Please address the information to the IETF Executive
   Director.

Full Copyright Statement

   Copyright (C) The Internet Society (2003).  All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by removing
   the copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the purpose of
   developing Internet standards in which case the procedures for
   copyrights defined in the Internet Standards process must be
   followed, or as required to translate it into languages other than
   English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assignees.

   This document and the information contained herein is provided on an
   "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
   TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
   BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION

Templin                  Expires August 1, 2003                 [Page 5]

Internet-Draft    MTU Issues in IPv6 Neighbor Discovery     January 2003

   HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
   MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Templin                  Expires August 1, 2003                 [Page 6]

---

From: Fred Templin <[email-masked]>
Date: Fri, 24 Oct 2003 14:50:59 -0700

Itojun,

I disagree with your statement. Two examples where the per-neighbor MTU
negotiation would be very useful:

 1) Peer-to-peer wireless (e.g. IEEE 802.11):
    On certain wireless media, receiver A can sense QoS metrics (e.g. SNR)
    for the packets it receives from senders B and C. If the SNR for B 
is strong,
    A may wish to inform B that a larger MTU (e.g. 1500 bytes) is 
acceptable.
    Conversely, if the SNR for C is weak, A may wish to inform C that a 
smaller
    MTU (e.g. 1300 bytes) is desired.

  2) Constrained nodes on large links:
    On large-MTU media (e.g., HiPPI, Gig-E, etc.), nodes with small 
receive buffers
    (e.g. boot ROMs, embedded devices) may wish to receive smaller 
packets than
    the MTU for the link. For example, a constrained node on a link with 
MTU of
    64KB may wish to inform a neighbor that an MTU of 10KB is desired.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Fri, 24 Oct 2003 14:57:59 -0700

>MTU's on the same link and want to optimize it, use multiple links..
Not possible in all cases; take multi-access wireless media, for example.

We must face facts that there are multiple access, shared-media links
out there and will continue to be for the forseeable future. We can wish
for a densely-connected mesh of point-to-point links (e.g., ATM PVCs)
but we needs-be have to support the L2 infrastructure that is already
deployed. (Otherwise, we would have to go through an L2 transition
before we even begin to tackle the IPv6 transition.)

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Fri, 24 Oct 2003 17:34:22 -0700

>length field (if I don't misremember). Thus a pair of nodes can agree to use

HiPPI is another example of a link with a 16-bit length field.
So, you can get up to 64KB MTUs but that might be too much
for constrained nodes to accept at high packet arrival rates. If
IPv6 ND provided a new option for nodes to send per-neighbor
(not per-link) MTU values, the situation for constrained nodes
on large links would be greatly improved.

>two hosts so that the packets no longer go through the same set of

If you are probing to determine the initial PMTU, the answer is to send
periodic additional probes, of course. Routes are going to flap, and paths
are going to change - a once-probed MTU cannot be expected to remain
stable for any set length of time.

>CRC for larger frames, the IEEE has been unvilling to look into this.
The IEEE transformation obviously just isn't going to happen. Even if it
somehow did, we would have the entire legacy Internet infrastructure to
overhaul and (as I said to Pekka) I just can't see waiting for an L2 
transition
before we move forward seriously with the IPv6 transition.

>would make sense to me. But without the L2 capability it doesn't seem

Disagree here. See my earlier examples on peer-to-peer wireless networking
and constrained nodes on large MTU media.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Fri, 24 Oct 2003 17:51:50 -0700

>"I can support receiving 9k".

Agree with the hosts sending ND messages saying:
"I can support receiving 9K" but not necessarily with the
rotuer saying "trust us; the L2 can in fact support 9k". The
router should be advertising the lowest-common-denominator
for all nodes on the link. This might be constrained by the link
with the smallest MTU, the node with the smallest receive
buffer, etc.

Nodes that can accept more than the lowest-common denominator
should negotiate a larger MTU with their neighbors - even if the
value they negotiate is larger than what is being sent in the MTU
option in RA's. (Similarly, nodes that prefer smaller than the LCD
should negotiate a smaller MTU - even though they still need to
support the LCD for broadcast/multicast.)

>1500 MTU for sending broadcast and multicast packets on the link.

Yes - this follows from the LCD discussion above.

>deciding that it should declare itself 9k capable.

You could use something like an IPv6 NS message that is artificially
inflated in size for this as we have discussed in the past. But, it's not
a once-and-done deal; if you're going to be probing the MTU you need
to do it periodically so that L2 path changes don't result in black holes.

General comment - it would be nice if folks would reveiw and comment
on my drafts:
 
  http://www.ietf.org/internet-drafts/draft-templin-ndiscmtu-00.txt
  http://www.ietf.org/internet-drafts/draft-templin-tunnelmtu-01.txt

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Fri, 24 Oct 2003 18:05:47 -0700

I see (at least) three ways for the neighbor-to-neighbor MTU negotiation;
two were presented in my drafts and the third is presented by Iljitsch here.
The methods are:

  1) Change RFC 2461 to allow NA messages to include MTU options:
    Advantages:
      - Unambiguous mechanism for NA's to inform of a per-neighbor MTU value
    Disadvantages:
      - Requires modification to RFC 2461
      - May require extra ND messages, since the interpretation of
        MTU options is different for RAs

  2) No changes to RFC 2461; allow "IPv6-over-foo" documents to
    specify different interpretations of the MTU option in RA's:
    Advantages:
      - No modifications to RFC 2461
    Disadvantages:
      - Ambiguous interpretation of MTU options in RAs, or the
      specified interpretation (MTU for the entire link) is disabled
      - MTU option only available for RAs; not NAs

  3) Change RFC 2461 to specify a new "NBR_MTU" option  that
    can be sent with either NA's or RAs:
    Advantages:
      -  Unambiguous mechanism to inform of a per-neighbor MTU value
      - Can be used with either NAs or RAs w/o altering the interpretation
        of the existing MTU option
      - Maximum efficiency, since it can be used with either NAs or RAs
    Disadvantages:
      - Requires modifications to RFC 2461

So, it should be clear from the above analysis that I support Iljitsch's
proposal in terms of what should go into RFC 2461. It is a sensible
approach for a valuable mechanism and I think folks should give it
the consideration it deserves.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 25 Oct 2003 19:10:31 +0200

>  http://www.ietf.org/internet-drafts/draft-templin-tunnelmtu-01.txt

Ok, the second one first:

As far as I can tell, not being someone who builds routers, some of the 
mechanisms outlined here are problematic. For instance, determining 
whether the IP packet that carries a tunneled packet was fragmented 
means transferring information from one place in the architecture 
(reception and reassembly of IP packets) to another (handling protocol 
41). Another: doing a router sollicitation triggered by wanting to 
transmit a packet of a certain size is not a good thing.

But why bother in the first place?

Presonally, I would happily let my tunnel packets be fragmented as this 
way I don't incur a reduced MTU when using a tunnel. Sure, this costs 
extra CPU time for the tunnel endpoints but in most cases this isn't a 
problem. And when it is, the tunnel endpoints should be able to do 
PMTUD over the tunnel. And if that doesn't work, it's always possible 
to configure a smaller MTU. Don't forget that tunnels are typically 
pretty static so once everything is set up, there shouldn't be too many 
surprises.

I think this draft is solving a non-problem.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 25 Oct 2003 23:53:00 +0200

On 25 okt 2003, at 2:51, Fred Templin wrote:

>> option that asserts "trust us; the L2 can in fact support 9k" 
>> resulting in individual hosts deciding to send out an option with 
>> their NS/NA saying "I can support receiving 9k".

Right, that's what I was talking about.

> for all nodes on the link.

It's already doing that today. The "L2 can do 9k" indication should go 
into a new message (option).

> option in RA's.

Yes, but they must be aware of what the layer 2 equipment can handle. 
If I hook up two boxes that can do 9000 bytes over ethernet, but the 
switch is 1500 bytes only, I had better make sure those two boxes stick 
to 1500.

> (Similarly, nodes that prefer smaller than the LCD

The good thing is that if we allow hosts to discover the maximum usable 
MTU between them, we can now use the regular RA MTU option to broadcast 
a much smaller MTU for this type of traffic.

>> deciding that it should declare itself 9k capable.

Who says the router can handle the maximum size packets on a link? In a 
SOHO environment it's not unusual to have some pretty fast boxes with 
gigabit ethernet that support 9000 bytes, but since the hookup to the 
internet is only a few megabits, the router has 100 Mpbs or even 10 
Mbps ethernet, with no support for jumboframes.

But an exchange using the advertised packet sizes between the two hosts 
wanting to use larger packets wouldn't be a bad idea.

> to do it periodically so that L2 path changes don't result in black 

ND times out pretty quickly, right? Then this happens pretty much 
automatically.

>  http://www.ietf.org/internet-drafts/draft-templin-ndiscmtu-00.txt

Well, you don't waste many words. This must be the shortest draft I've 
ever seen.

You only consider the situation where the per-neighbor MTU is smaller 
than the "official" link MTU as defined by the relevant "IPv6 over ..." 
RFC or the MTU option in router advertisements.

However, the situation where some hosts have the ability to use larger 
packets than others and/or the official MTU could also easily be 
addressed by the same mechanism. However, in that case there is the 
additional complication that devices operating at lower layers (so that 
they're unable to participate in IP layer mechanisms) may restrict the 
maximum packet size. Now hopefully at some point layer 2 mechanisms to 
deal with this will become available, but in the mean time we need to 
solve this in IP. (I can imagine adding information about the MTU to 
the ethernet autonegotiation protocol.)

The way I see it, we can't touch the existing RA MTU option, because 
that would lead to trouble with hosts that don't understand the new way 
of handling the MTU. So the current RA MTU option would continue to be 
used to advertise the minimum MRU (maximum receive unit) that all hosts 
on the link must support. This value must be somewhere between 1280 
bytes and the value listed in the IP over RFC.

Then there would be a new RA option that would function as the "maximum 
MTU": hosts are not allowed to transmit packets larger than this. This 
value must be equal to or higher than the IP over RFC value if it's 
present. There could be a bit in this option that indicates "if lower 
layers tell you it's ok to use this value" or "forget what lower layers 
tell you, this value is the correct one".

Finally there would be the per-neighbor MTU preference/capability 
option. If the advertised neighbor MTU is smaller than the link MRU 
this indicates that the neighbor is still able to receive packets upto 
the size of the link MRU, but it prefers to receive smaller packets. If 
the neighbor MTU is larger than the link MRU the neighbor is capable of 
and willing to receive packets larger than the link MRU. In that case, 
if the local system has a similar capability and willingness, it may 
send packets of a size that is the minimum of the local interface MTU, 
the link MMTU and the MTU advertised by the neighbor.

And if we're going to do this anyway, we may want to provide for a way 
for routers to tell hosts what the largest packet size is that it can 
forward. For instance, a simple IPv6-capable ADSL router obviously 
supports a 1500 byte MTU on its ethernet interface, but if it uses a 
tunnel for IPv6 connectivity it can't forward IPv6 packets that are 
larger than 1480 bytes. So advertising this 1480 byte limit in a RA 
option saves hosts from hitting PMTUD on the first hop all the time. 
Also, if there is more than one router on the link and they support 
different maximum routable packet sizes, the host can select the router 
that supports the largest packets.

Back to your draft. I would suggest making the draft a self-contained 
set of new capabilities rather than going for updates of RFC 2461. This 
should make both the technical and political aspects much less painful.

(Adding a new ND option type is an obvious IANA cosideration, BTW.)

Iljitsch van Beijnum

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 12:57:17 +0100 (CET)

> stable for any set length of time.

Did you really mean that?
Clearly there has to be some assumption about the frequency at which
the MTU can decrease. If not, for every e.g. TCP packet you send you would need
to send an MTU probe (with an ACK) to make sure that a (TCP) packet of that
size can make it to the neighbor.

And since this is the L2 MTU and there will be multiple L2 instances along the
L3 path one would either need to do such probing end2end (doubling the number
of packets on the Internet), or have routers on the path perform hop-by-hop
MTU probing including from the last hop router towards the destination host.

   Erik

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 13:10:07 +0100 (CET)

> ND times out pretty quickly, right? Then this happens pretty much 

FWIW RFC 2461 does not have any timeout associated with the neighbor cache
entries or destination cache entries.
They can be garbage collected when the node runs low on memory.
And stale link-layer addresses in the neighbor cache are detected when
the link-layer address is used by triggering Neighbor Unreachability Detection.
 Thus if you'd want some mechanisms to peridically "reprobe" the path to the
neighbor you'd have to explicitly add such a mechanism.

  Erik

---

From: Pekka Savola <[email-masked]>
Date: Mon, 27 Oct 2003 14:48:07 +0200 (EET)

>     MTU (e.g. 1300 bytes) is desired.

This kind of dynamic MTU negotiation (requires at least one round-trip!) 
seems rather complex to me.  Even more so that the original proposal 
seemed to be ("node A advertises that a larger MTU is OK").

>     64KB may wish to inform a neighbor that an MTU of 10KB is desired.

Are you sure such deployments are feasible to begin with?  Such low-end 
equipment is very unlikely equipped with e.g. Gig-E interfaces.  (Note: 
Gig-E MTU maximum is about 9K already, so it's a bit more reasonable 
than 64K.)

(from Fred's another message on the list:)
> before we even begin to tackle the IPv6 transition.)

Sure.  There are tradeoffs in every kind of media.  A typical assumption
for most kinds of multi-access shared-media links is that they have
similar properties regarding MTU.  This is also the case for larger MTU's.  
This problem is nowhere near non-trivial.  Consider e.g. an IPv6 link
which is separated by two Ethernet switches, one which supports
jumboframes (MTU=9K) and one which does not.  How would the hosts be able
to, with simple and robust mechanisms, to determine the MTUs to be used on 
such environment?

The dynamic, wireless multiple-access media seems to have even more
challenging problems of having to probe and adjust MTU's all the time.  
Think about an application using UDP (ie., has to do optimize for the MTU
value itself) that runs longer than the lifetime of this probing.  You
expect these to adjust all the time as well?

This approach seems to be adding complexity of L2 to IP layer where it 
doesn't belong.  If we need to tackle this issue at L3, why not just 
simply use MTU 1280 or MTU 1500 and fragment the packet when needed?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 27 Oct 2003 19:34:14 +0100

On 27 okt 2003, at 13:48, Pekka Savola wrote:

>> If the SNR for B is strong, A may wish to inform B that a larger MTU 
>> (e.g. 1500 bytes) is acceptable. Conversely, if the SNR for C is 
>> weak, A may wish to inform C that a smaller MTU (e.g. 1300 bytes) is 
>> desired.

> seemed to be ("node A advertises that a larger MTU is OK").

There is no need for any full-blown negotiation. A node can simply 
announce the desired/supported maximum packet size at the time of 
neighbor discovery and be done with it.

>> 2) Constrained nodes on large links: On large-MTU media (e.g., HiPPI, 
>> Gig-E, etc.), nodes with small receive buffers (e.g. boot ROMs, 
>> embedded devices) may wish to receive smaller packets than
>>     the MTU for the link.

> Gig-E MTU maximum is about 9K already, so it's a bit more reasonable

Hm, if we look back 15 years we see that hosts needed to support 1500 
bytes for 10 Mbps ethernet. A typical example of such a host was a PC 
with less than 1 MB memory. Today networks are 100 times as fast (GE is 
now cheaper than 10 Mbps 15 years ago), we typically have 500 times as 
much memory but our maximum packet size has increased by a whopping six 
times...

I agree that it is unlikely a host with GE or similar would be unable 
to support a decent maximum packet size, but it can't hurt to have the 
option of triggering a smaller packet size.

> A typical assumption
> similar properties regarding MTU.

Shared media? That is sooo last century...

> This is also the case for larger MTU's.

This is why we need routers to explicitly advertise the maximum MTU 
using a new option. It is unlikely that people who need the extra 
performance gained by the use of jumboframes are also the ones that 
hook up a $20 switch to a $600 one.

It wouldn't hurt if the IEEE picked this up, though.

> challenging problems of having to probe and adjust MTU's all the time.

802.11 does this at layer 2, which seems the right thing to do.

> simply use MTU 1280 or MTU 1500 and fragment the packet when needed?

In IPv6, fragmentation overhead is too hideous and probably not 
supported below 1280 bytes without changes anyway.

---

From: Pekka Savola <[email-masked]>
Date: Mon, 27 Oct 2003 20:53:03 +0200 (EET)

> neighbor discovery and be done with it.

Some seem to want a full-blown option as well.  Whether there would 
consensus to support it would be a different topic..

> option of triggering a smaller packet size.

If such nodes are not expected to be common, it would seem to make sense
to separate these very constrained nodes to a separate subnet (not
necessarily even a separate link) if the full MTU size seems like a 
lucrative option.

> > This is also the case for larger MTU's. This problem is nowhere near
> > non-trivial.  Consider e.g. an IPv6 link which is separated by two
> > Ethernet switches, one which supports jumboframes (MTU=9K) and one
> > which does not.  How would the hosts be able to, with simple and
> > robust mechanisms, to determine the MTUs to be used on such
> > environment?
> 
> using a new option. 

No need for that, already supported now :-).

> It is unlikely that people who need the extra 

You would be surprised which vendors don't support jumbograms.  The bit 
more expensive ones as well, for example, HP (very widely used around 
here at least).
 
> supported below 1280 bytes without changes anyway.

But if this was only needed for very constrained nodes, one could argue
that such nodes probably don't need do heavy communication anyway
(otherwise they wouldn't be constrained nodes :-), so this shouldn't be a
problem.. :-)

---

From: Fred Templin <[email-masked]>
Date: Mon, 27 Oct 2003 11:50:41 -0800

Iljitsch van Beijnum wrote:

> As far as I can tell, not being someone who builds routers, some of 
> the mechanisms outlined here are problematic. For instance, 
> determining whether the IP packet that carries a tunneled packet was 
> fragmented means transferring information from one place in the 
> architecture (reception and reassembly of IP packets) to another 
> (handling protocol 41).

(We are talking here about the specific instance of IPv4 used as a 
link-layer
for IPv6.) Your point about instrumenting the IPv4 reassembly code is well
taken. But, this is just one option and perhaps not the best for all 
architectues.

For example, Linux offers a "packet socket" abstraction ("man 7 packet")
whereby an application can capture selected L2 packets based on filters.
It is a rather trivial thing to write a Linux packet socket filter that 
captures
all IPv4 packets with fragment headers for ip-proto-41 w/o having to touch
a line of kernel code. The only arguement then becomes one of performance,
since ip-proto-41 fragmented packets would be handled at the application
level. But, the protocol is designed to naturally minimize the number of
fragmented packets for ip-proto-41 so fast-path processing is not necessary.

> Another: doing a router sollicitation triggered by wanting to transmit 
> a packet of a certain size is not a good thing.

You've gotten wires crossed between Method 1 (section 3.3.1) and Method 2
(section 3.2) in my draft. Method 1 uses fragmentation sensing in the 
receiver
as the method for detecting path MTU changes and does not require the
encapsulator to send an RS triggered by wanting to send a packet of a 
certain
size. In the fragmentation sensing method, each data packet also serves as a
probe packet w/o having to introduce any additional messages. (Method 2
does suggest on-demand probing based on the arrival of large packets, but it
is intended only for those circumstances in which the fragmentation-sensing
scheme is not supported.)

> Presonally, I would happily let my tunnel packets be fragmented as 
> this way I don't incur a reduced MTU when using a tunnel.

Yes, *you* might be happy about letting your tunnel packets be fragmented,
but would the *network* be happy about it? Certainly not. A quick read of
"Fragmentation Considered Harmful" and "Beyond Folklore: Observations
on Fragmented Traffic" should easily convince you of this. See:

  http://research.compaq.com/wrl/techreports/abstracts/87.3.html
  http://www.caida.org/outreach/papers/2002/Frag/

> Sure, this costs extra CPU time for the tunnel endpoints but in most 
> cases this isn't a problem.

See the above documents. It isn't so much the CPU time for the tunnel
endpoints we are concerned with as the case of fragments lost due to
congestion which results in a loss unit (a packet fragment) that is smaller
than the retransmission unit (a  packet). So, I take back my statement above
that "you" might be happy about letting  your tunneled packets be fragmented
since the original source of the large packets will also suffer in the 
long run.

> And when it is, the tunnel endpoints should be able to do PMTUD over 
> the tunnel.

IPv4 PMTUD you mean? If so, please see the second paragraph in the
introduction of my document for why this is undesireable in terms of
efficiency and robustness.

> And if that doesn't work, it's always possible to configure a smaller MTU.

Well, the smallest static MTU we can assign for  IPv6-in-IPv4
tunnels is 1280 bytes, since 1280 is the min MTU for IPv6 interfaces.
Even at that size, however, it is still possible that we might encounter
IPv4 paths that will fragment the 1280byte packets due to, e.g., too
many nested layers of encapsulation.

> Don't forget that tunnels are typically pretty static so once 
> everything is set up, there shouldn't be too many surprises.

The only static thing about the tunnels is the endpoints. The intervening
IPv4 path may be arbitrarily long and dynamically fluctuating. That's
why path MTU discovery can never be a once-and-done deal but must
be continuous throughout the tunnel's operational lifetime.

> I think this draft is solving a non-problem. 

Please re-read and reconsider in light of the above.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Mon, 27 Oct 2003 15:12:37 -0800

Iljitsch van Beijnum wrote:

 >>> So one approach would be something like having a router
 >>> advertisement option that asserts "trust us; the L2 can in fact
 >>> support 9k" resulting in individual hosts deciding to send out an
 >>> option with their NS/NA saying "I can support receiving 9k".
 >>
 >

 > switch is 1500 bytes only, I had better make sure those two boxes
 > stick to 1500.

Yes, I know. By "negotiate a larger MTU" I mean not only an
initial indication of how much the *neighbor* can handle but
also ongoing and continuous attention to how much the  *L2
media* can handle.

 > The good thing is that if we allow hosts to discover the maximum
 > usable MTU between them, we can now use the regular RA MTU option to
 > broadcast a much smaller MTU for this type of traffic.

But, if there are many routers on the link, what do you do if
they aren't all broadcasting a consistent smaller MTU? RFC
2461 says to accept the most recently heard MTU - is that
good enough?

 > Who says the router can handle the maximum size packets on a link? In
 > a SOHO environment it's not unusual to have some pretty fast boxes
 > with gigabit ethernet that support 9000 bytes, but since the hookup to
 > the internet is only a few megabits, the router has 100 Mpbs or even
 > 10 Mbps ethernet, with no support for jumboframes.
 >
 > But an exchange using the advertised packet sizes between the two
 > hosts wanting to use larger packets wouldn't be a bad idea.

Agreed.

 > Well, you don't waste many words. This must be the shortest draft I've

Economy and effectiveness in written expression are ideals
I continue to strive for.

 > than the "official" link MTU as defined by the relevant "IPv6 over
 > ..." RFC or the MTU option in router advertisements.
 >
 > additional complication that devices operating at lower layers (so
 > that they're unable to participate in IP layer mechanisms) may
 > restrict the maximum packet size. Now hopefully at some point layer 2
 > mechanisms to deal with this will become available, but in the mean
 > time we need to solve this in IP. (I can imagine adding information
 > about the MTU to the ethernet autonegotiation protocol.)
 >
 > that would lead to trouble with hosts that don't understand the new
 > way of handling the MTU. So the current RA MTU option would continue
 > to be used to advertise the minimum MRU (maximum receive unit) that
 > all hosts on the link must support. This value must be somewhere
 > between 1280 bytes and the value listed in the IP over RFC.

Agreed.

 > Then there would be a new RA option that would function as the
 > "maximum MTU": hosts are not allowed to transmit packets larger than
 > this. This value must be equal to or higher than the IP over RFC value
 > if it's present. There could be a bit in this option that indicates
 > "if lower layers tell you it's ok to use this value" or "forget what
 > lower layers tell you, this value is the correct one".

I'm afraid I'm not bought into this one as being necessary (yet). We know
from our physical/logical point of attachment what the largest possible
MTU for the attached L2 media is - this is a given. So, to supply a maximum
MTU that is larger than the attached L2 media can support in a single packet
is saying that we expect the sender to do L2 fragmentation locally. Is this
what you are saying, and is this really desireable? (Maybe; I'm willing to
be convinced.)

 > the size of the link MRU, but it prefers to receive smaller packets.
 > If the neighbor MTU is larger than the link MRU the neighbor is
 > capable of and willing to receive packets larger than the link MRU. In
 > that case, if the local system has a similar capability and
 > willingness, it may send packets of a size that is the minimum of the
 > local interface MTU, the link MMTU and the MTU advertised by the neighbor.

Sure.

 > different maximum routable packet sizes, the host can select the
 > router that supports the largest packets.

Right - this goes along with a question I asked above.

 > set of new capabilities rather than going for updates of RFC 2461.
 > This should make both the technical and political aspects much less
 > painful.
 >

OK.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 28 Oct 2003 00:26:41 +0100

> No need for that, already supported now :-).

No, the existing option is only usable for lowering the MTU used within 
a subnet to a value lower than what the "IP over ..." RFC prescribes. 
We also need a way to convey the maximum packet size that layer 2 
equipment can handle. I think this should be an option that is similar 
to, but different from the existing MTU option.

> more expensive ones as well, for example, HP (very widely used around

Well in that case you have nothing to worry about and stick to 1500 
bytes.  :-)

---

From: Fred Templin <[email-masked]>
Date: Mon, 27 Oct 2003 15:34:04 -0800

>size can make it to the neighbor.

No, we don't want to send an additional probe packet along with
*every* data packet. If we can somehow use the data packets
themselves as probe packets, then that would be a different
story of course.

One case in which data packets could be used as probe packets
is when IPv4 is used as an L2 media for IPv6. In this case, we could
send IPv6-in-IPv4 encapsulated packets with the DF bit NOT set
in the IPv4 header expecting that the decapsulator would send us
some sort of indication if it sensed fragmentation.

But this begs the question of a fundamental design point: do we need
to support sub-L2 media elements (i.e., the physical elements that sit
below IPv4) that neither support IPv4 fragmentation nor send IPv4
"frag needed" messages when they can't forward a packet? Based on
the 1Gbps/100Mbps Ethernet bridge example, I believe the answer
to this is "yes" - would you agree?

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Mon, 27 Oct 2003 16:20:01 -0800

>seemed to be ("node A advertises that a larger MTU is OK").

No; the proposal is that node A initially advertises to B and C the largest
packet size it would like to receive from each of them. A then sits back and
passively accepts packets from B and C until it senses some change in 
conditions
(e.g., B's SNR degrades or C's SNR improves). A can then send an 
*unsolicited*
message to B and/or C to inform that a new maximum packet size is desired.
(A should of course give some consideration to rate limiting the unsolicited
messages it sends.)

>Gig-E MTU maximum is about 9K already, so it's a bit more reasonable 

Well, every node on a link should at least be able to receive a packet as
large as the MTU of the link. But, some nodes might prefer to receive
smaller packets and so should have some means of telling neighbors that
smaller packets are desired.

>to, with simple and robust mechanisms, to determine the MTUs to be used on 

You are entering the realm of bridging L2 media with diverse MTUs;
IPv6/IPv4 path MTU discovery doesn't work in this case unless the
bridges are "transluscent" and send ICMP "packet too big" messages
as they drop the too-big packets. (Note that a "transparent" bridge would
not look into the L3 header and simply drop the too-big packet w/o
sending any  error message at all.)

Can we augment path MTU discovery to work with transparent bridges
that connect media with diverse MTUs? Seems like we need to - whether
the links are multi-access shared media or point-to-point.

>expect these to adjust all the time as well?

As I said above, consideration to rate limiting the unsolicited MTU
change messages needs to be given in order to dampen the frequency.

>simply use MTU 1280 or MTU 1500 and fragment the packet when needed?

We will miss out on the opportunity to take advantage of larger MTUs
if we set a static MTU and just let IP fragment the packets. Also, if we
have transparent bridges in the path we won't have the required IP
fragmentation support at all forwarding nodes and we'll get black holes.

Fred
[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 28 Oct 2003 09:55:21 +0100

> also ongoing and continuous attention to how much the  *L2

But unfortunately this isn't something that can easily be determined. 
Is it ok to just send large probe packets, or can sending packets that 
are too large be harmful?

> they aren't all broadcasting a consistent smaller MTU?

Beat the administrator.

> RFC 2461 says to accept the most recently heard MTU - is that
> good enough?

I'd say: trust the router you've chosen as your default gateway. I 
considered picking the lowest from those advertised, but this would 
make it trivial to do "MTU reduction attacks".

> MTU for the attached L2 media is - this is a given.

Yes, but who is "we"? The administrator knows this after perusing the 
documentation and obviously the box itself knows at some level. We 
currently don't have any mechanisms to transfer this knowledge to the 
layer 3 on hosts that are considering using jumboframes.

The IEEE was understandably reluctant to increase the gigabit ethernet 
MTU beyond 1500 bytes because this would break interoperation with 
existing 10 and 100 Mbps ethernet. But at some point something has to 
give, because two hosts that want to utilize 10 gigabit ethernet 
between them would have to transmit/receive almost a million packets 
per second with a 1500 byte MTU.

> So, to supply a maximum
> is saying that we expect the sender to do L2 fragmentation locally.

Obviously announcing an MTU that's larger than what layer 2 can support 
doesn't make sense. But often layer 2 can support a larger MTU than 
what is specified for that particular layer 2 protocol and/or for IP 
over that protocol. This "unofficial" MTU is the one we're interested 
in. (At least when increasing the MTU.)

> Is this what you are saying, and is this really desireable? (Maybe; 
> I'm willing to be convinced.)

If layer 2 fragmentation is done there should also be layer 2 
reassembly. Then the whole procedure becomes transparent to IP so we 
don't have a problem. This is what 802.11 does to achieve interference 
robustness. AAL5 in ATM is basically the same thing.

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 28 Oct 2003 11:36:58 +0100 (CET)

> to, but different from the existing MTU option.

Just to restate what you are saying differently so that others might understand
it. Today with have a MTU specified in the IPv6 over foo documents.
We have an MTU option in router advertisements that can be used to
lower that number - which is useful(?) for the last century problem of
bridging Ethernet and FDDI together with half-broken fragmenting
bridges - can get the FDDI attached nodes to use 1500.

What Iljitsch is sayin is that if the L2 might support a larger MTU
e.g. by ensuring that all the switches support Ethernet jumboframes
one could add an option (which he calls "MAX MTU" - "maximum L2 packet size"
might be a more differentiated term).
But if the routers advertised this option, it still wouldn't mean
that a given host was capable of receiving L2 packets of that size.
Hence the need for each host to advertise their MRU in NS/NA type
packets.

I think this works as long as the network admin can ensure that all the
switches in an L2 in fact support the larger size.

Whether it is important to solve this using additional protocol mechanisms,
or whether it is sufficient to have folks hand-craft VLANs containing
the hosts that support the larger size, is hard to tell.

  Erik

---

From: Jari Arkko <[email-masked]>
Date: Tue, 28 Oct 2003 17:27:17 +0200

Erik Nordmark wrote:

 > might not support jumboframes.

I'd favor robustness a lot more than optimization for the highest
available MTU size. Whatever we do, lets make sure the protocol
can deal with the above case.

 > One way to cope with this particular topology,

Yes. But somehow I'm worried about this, particularly when
the MTU size field in ND is 32 bits. Is there any danger that
a false claim of a large MTU size will lead to something
bad happening? Or are we relying on the sender's hardware
to not accept overly large packets for transmission?

Also, if the IP header is 40 bytes/packet, one exchange
of two 9k packets consumes as much bandwidth as
the overhead from 225 packets -- a 337 K transmission
at 1500 byte MTU. So perhaps you wouldn't like to do this
every time.

--Jari

---

From: Fred Templin <[email-masked]>
Date: Tue, 28 Oct 2003 10:35:10 -0800

> are too large be harmful?

Well, there's no use in sending large probe packets if you aren't going
to allow sending large data packets. The purpose of the probes is to
determine a window of opportunity during which it should  be OK
to send large data packets.

>> they aren't all broadcasting a consistent smaller MTU?

I  wish!

> layer 3 on hosts that are considering using jumboframes.

By "we", I was referring to the interface that is configured over the L2 
link.
In the case of Ethernet - I hear you and Erik telling me that the 
interface has
no way of knowing whether the physical link is capable of supporting an
MTU of 9KB for jumboframes or only 1500bytes - do I have this correct?

> per second with a 1500 byte MTU.

So, you are saying that the IEEE was reluctant to increase the Gig-E MTU
beyond 1500 bytes. The MTU speaks only to the *send* side - but, what
about the MRU (receive side)? Can Ethernet interfaces determine whether
they are configured over a Gig-E link and (if so) make sure the MRU is
at least 9KB?

> Obviously announcing an MTU that's larger than what layer 2 can 
> support doesn't make sense. But often layer 2 can support a larger MTU 
> than what is specified for that particular layer 2 protocol and/or for 
> IP over that protocol. This "unofficial" MTU is the one we're 
> interested in. (At least when increasing the MTU.)

Well, if an L2 infrastructure has a mix of 10/100 and Gig-E
Ethernet elements then I suppose routers on the L2 could advertise
Max/Min MTU values such that:

  1280 <= Min_MTU <= 1500 <= Max_MTU <= 9KB

Then, each neighbor could announce a per-neighbor MTU
such that:

  1280 <= NBR_MTU <= Max_MTU

Do I have this right?

> robustness. AAL5 in ATM is basically the same thing.

Yes, if the L2 supports fragmentation then the L3 can send and receive
whole packets transparently - but this should only be allowed up to the
point that triggering the L2 fragmentation does not impart issues such as
excessive fragment loss due to congestion.

See sections 3.4 ("Use of Transparent Fragmentation") and 3.5 ("Careful
use of Intentional Fragmentation) of "Fragmentation Considered Harmful"
for  more thorough discussion on this.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Tue, 28 Oct 2003 11:11:07 -0800

>bridges - can get the FDDI attached nodes to use 1500.

These are what I referred to as "transluscent bridges" in an earlier
message. Perhaps that term (circa 1988/89) was only used within
certain limited enclaves, e.g., Digital Equipment Corporation.

>might be a more differentiated term).

I don't agree that ensuring that all switches support Ethernet
jumboframes is a pre-requisite for routers to advertise a larger
Max_MTU. Even if only *some* of the switches support the
jumboframes, it should be OK as long as...

>Hence the need for each host to advertise their MRU in NS/NA type

...as long as the hosts *also* do this.

>switches in an L2 in fact support the larger size.

Well, I just don't see this as realistic except in certain constrained
enclaves. So, even if we have each host advertising its MRU (e.g.,
in NS/NA type packets) we still have to verify that packets of
that size actually arrive and w/o incurring harmful levels of L2
fragmentation. Also, since L2 paths can fluctuate, we would need
to re-verify periodically.

>the hosts that support the larger size, is hard to tell.

I fall on the side of the former. Even if folks hand-craft the
VLANs, they can still get instances of link MTU reductions
in the L2 path between any pair of hosts unless they have
complete assurance that all of the switches in the path support
the larger MTU - and, that the L2 paths can't fluctuate such
that links with smaller MTUs are incorporated.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Tue, 28 Oct 2003 11:21:17 -0800

> to not accept overly large packets for transmission?

False claims can be mitigated by employing mechanisms to
authenticate ND messages, e.g., SEND.

> of two 9k packets

It is not an exchange of two 9K packets; it is a 9K probe
packet followed by a much smaller acknowledgement
packet - on the order of the size of a NA message. The
MTU/MRU probing is unidirectional.

> consumes as much bandwidth as

I didn't quite catch this - can you re-phrase?

Fred
[email-masked]

---

From: Jari Arkko <[email-masked]>
Date: Wed, 29 Oct 2003 07:07:33 +0200

> authenticate ND messages, e.g., SEND.

SEND can assure that RAs come from an authorized router,
and that NAs come from the owner of an address. SEND may
not be able to assure that the sender of an NA is trusted.
That is, we know it comes from the node in question, but
we may not be able to trust all the parameters it sends.
In conclusion if the MTU info comes from a router, SEND
is sufficient, but if its host to host, SEND may not be
sufficient.

Anyway, I still think this danger is _mostly_ a non-issue --
your regular 10 Mbit/s ethernet hardware would hopefully decline to
send a 2^32 byte probe packet and spend 3400 seconds sending
it. However, there might be some special network technology
where this might be an issue. Anyway, anything beyond 2^16
would require a jumbogram.

> I didn't quite catch this - can you re-phrase?

I'm assuming the reason for choosing a larger MTU is to
save in header overhead and processing time. But the savings
in header overhead must be balanced against the cost of
negotiating a larger MTU, particularly if that negotiation
involves sending large probe packets.

So, one 9K probe is about the same size as the overhead
from extra 225 IPv6 headers. Using the standard 1500 byte
MTU you get to send about 337 K before spending this much
in overhead. That is, a 9K probe packet does not make sense
bandwidth-wise if you are communicating less than 337 K
with your peer.

--Jari

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 29 Oct 2003 16:34:43 +0100 (CET)

> to not accept overly large packets for transmission?

Presumably a stack that implements such a future capability would
need to verify that the driver/NIC don't fail if they are fed
a too large packet.

> bandwidth-wise if you are communicating less than 337 K

That is in terms of bandwidth overhead.
I think the strongest motivation for jumboframes is that it reduces
software overhead in the hosts with as much as a factor of 6 (due
to having to deal with one sixth as many packets to transfer the
same amount of bulk data).

  Erik

---

From: Fred Templin <[email-masked]>
Date: Wed, 29 Oct 2003 09:48:43 -0800

> is sufficient, but if its host to host, SEND may not be

Well, maybe we should only accept MTU negotiations from
routers then; at least that's better than nothing at all.

> would require a jumbogram.

Agree in terms of the danger being _mostly_ a non-issue. We can
use SEND to ensure that RAs indeed come from an authorized router
(thanks for clarifying this) so we don't run the danger of sending
too-large or too-small packets to final destinations for which the
authorized router serves as next-hop.

In the case of host-to-host, the worst that can happen when SEND is
used is that the receiving host could misinform the sender of the MTU
size. Whether the bad size is too small or too large, in the long run it
is only the receiver itself (i.e., the sender of the bad MTU information)
that will suffer.

> bandwidth-wise if you are communicating less than 337 K

Sure; and it only gets worse for links with larger MTUs to probe.

Well, that about wraps it for me - I guess we'll just go ahead and
use the data packets themselves as probes then.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Wed, 29 Oct 2003 17:43:41 -0800

I seem to have gotten my wires crossed and thought the I-D cutoff
date was October 31st. No big deal; here are a couple of draft updates
relating to this subject thread that I will be submitting when the I-D
Registrar re-opens for business:

  http://www.geocities.com/osprey67/ndiscmtu-01.txt
  http://www.geocities.com/osprey67/tunnelmtu-02.txt

Comments welcome,

Fred Templin
[email-masked]

---

From: Mark Smith <[email-masked]>
Date: Thu, 30 Oct 2003 23:23:42 +1030

Hi All,

Came across this when looking at some other MTU related info.

http://www.psc.edu/~mathis/MTU/#pmtud

There is an ID at

http://www.psc.edu/~mathis/MTU/draft-mathis-plpmtud-00.txt

The BoF slides are at

http://www.psc.edu/~mathis/papers/mtuBOF200303/index.html

Hope this helps,
Mark.

---

From: Fred Templin <[email-masked]>
Date: Thu, 30 Oct 2003 09:54:58 -0800

Mark,

Many of us have known about this work for a long time now. If
there are certain aspects of it you would like to call to our attention
(e.g., if there is a way for the lower layers to know when the upper
layers are doing PLPMTUD) please send specific details. In the
meantime, I will be turning my attention to other topics.

Thanks - Fred
[email-masked]

P.S. The draft you cited is now obsoleted by
    'draft-ietf-pmtud-method-00.txt'.