Subject: jumbo frame of GbE and IPv6

From: Ryota Hirose <[email-masked]>
Date: Wed, 20 Jul 2005 18:37:54 +0900 (JST)

Hi IPv6 hackers,

Recent GbE NICs or GbE switches support Jumbo Frames, but RFC2464
provides that the maximum MTU of an Ethernet is 1500.  So we cannot
use the Jumbo Frames for IPv6.

2.  Maximum Transmission Unit

   The default MTU size for IPv6 [IPV6] packets on an Ethernet is 1500
   octets.  This size may be reduced by a Router Advertisement [DISC]
   containing an MTU option which specifies a smaller MTU, or by manual
   configuration of each node.  If a Router Advertisement received on an
   Ethernet interface has an MTU option specifying an MTU larger than
   1500, or larger than a manually configured value, that MTU option may
   be logged to system management but must be otherwise ignored.

For example, on FreeBSD 5.4R, to send a Jumbo Frames of IPv4, I just
only set the link MTU with ifconfig command.  To send a Jumbo Frames
of IPv6, I had to hack a kernel code to not compare link MTU with
ETHERMTU(1500).

   BTW, Jinmei san suggested that KAME, NetBSD and OpenBSD are already
   hacked as above and this behavior may be only FreeBSD problem.
   But, according to RFC2464, FreeBSD behavior is right, and other
   BSDs is wrong.

I know that this Jumbo Frames of GbE violates a specification of IEEE
802.3.  But it is deploied already, and is used for a high performance
application like the file servers.  Does anyone have a plan to change
the rule of RFC2464?

Ryota Hirose
Yamaha Corporation

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Hugh LaMaster <[email-masked]>
Date: Wed, 20 Jul 2005 10:58:20 -0700

> use the Jumbo Frames for IPv6.

As I read it, RFC2464 says that the maximum *default* MTU is 1500,
and, anything larger than that must be manually configured.  It
doesn't say that it is prohibited.

>    But, according to RFC2464, FreeBSD behavior is right, and other

I haven't tried this myself on FreeBSD 5.4R, but, if so, it is
a bug, because manual configuration of MTU larger than 1500 is
not prohibited by RFC2464.  Thanks for pointing the FreeBSD bug out.
Please report it to the FreeBSD mailing lists (-current, -net).

> application like the file servers.  Does anyone have a plan to change

Jumbo frames are essential, and, do not violate RFC2464.
Unfortunately, they are effectively deprecated by requiring
manual configuration of each node, since RFC2464 effectively
disallows configuration of jumbo frames through router
advertisements.

Probably OK, though, since most jumbo frame subnets today
are manually configured anyway.  However, the RFC
should not discourage jumbo frames.

---

From: Ryota Hirose <[email-masked]>
Date: Thu, 21 Jul 2005 08:01:52 +0900 (JST)

Hi,

>From: Hugh LaMaster <[email-masked]>
>Date: Wed, 20 Jul 2005 10:58:20 -0700

> doesn't say that it is prohibited.

RFC2464 says as following:

             This size may be reduced by a Router Advertisement [DISC]
    containing an MTU option which specifies a smaller MTU, or by manual
    configuration of each node.

I'm sorry for my poor English knowledge, but I think this sentence
permit to decrease MTU from default size by manual configuration.

OK, this sentence or others in the section don't prohibit increase
MTU.  Since decrease MTU is permitted and there is no mention to
increase, I understood that the increase MTU is prohibited.  Isn't it
a common sense?

Please refer RFC2467, which is about FDDI.  FDDI's MTU is variable by
network configuration, but RFC2467 provides the default MTU is 4352
for some reasons.  After that, same sentences with RFC2464 follow, but
I cannot read as larger MTU than 4352 is permitted.

And RFC2461, which is about ND and RS/RA, says about host behavior
which receive RA,

   If the MTU option is present, hosts SHOULD copy the option's value
   into LinkMTU so long as the value is greater than or equal to the
   minimum link MTU [IPv6] and does not exceed the default LinkMTU value
   specified in the link type specific document (e.g., [IPv6-ETHER]).

This is prohibited that MTU option exceed the *default* LinkMTU.

> >    But, according to RFC2464, FreeBSD behavior is right, and other

In my environment, Windows XP SP2 accepts larger MTU than 1500 and
sends Jumbo Frames.  Of course, if the implementation want to use
Jumbo Frames, it will have to accept large MTU.  Many implemetations
work so, and FreeBSD's behavior is a bug from the point of view of
users.  But, according to RFC2464 and RFC2461 strictly, I think we
cannot use Jumbo Frames on GbE.  Is an update memorandum about Jumbo
Frames necessary, isn't it?

Ryota Hirose
Yamaha Corporation

---

From: Ryota Hirose <[email-masked]>
Date: Thu, 21 Jul 2005 08:16:08 +0900 (JST)

Hi,

>From: Ryota Hirose <[email-masked]>
>Date: Thu, 21 Jul 2005 08:01:52 +0900 (JST)

> Please refer RFC2467, which is about FDDI.  FDDI's MTU is variable by

Please refer RFC2497 also. It is about ARCnet.  ARCnet's maximum MTU
is 60480, but RFC2497 says that it is impractical, so provides that
default MTU for IPv6 is 9072.  And says,

   In the presence of a router, this size MAY be changed by a Router
   Advertisement [DISC] containing an MTU option. If a Router
   Advertisement is received with an MTU option specifying an MTU larger
   than 60480, or larger than a manually configured value less than
   60480, that MTU option may be logged to system management but MUST be
   otherwise ignored.

RFC2497 use a word 'be changed' instead of 'be reduced', because
ARCnet can increase MTU to 60480.

Ryota Hirose
Yamaha Corporation

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 21 Jul 2005 10:19:56 +0200

> increase, I understood that the increase MTU is prohibited.  Isn't it

Well, that's a philosophical question: is everything that isn't  
explicitly disallowed allowed by default, or is everything that isn't  
explicitly allowed disallowed by default?

My common sense tells me that the authors of RFC 2464 didn't consider  
the case where the MTU would legitimately be larger than 1500 bytes.  
They did consider the case where router advertisements contain an MTU  
that is apparently incorrect, because it's larger than the standard  
allows.

Ideally, the MTU would be a neighbor discovery option, so that the  
highest possible MTU can be used between two systems on a subnet,  
without requiring that all systems on a subnet support the same MTU  
size. However, this is problematic because switches and hubs also  
have MTU limitations that can't be discovered this way.

In any event, RFC 2464 doesn't say it's disallowed to configure an  
MTU larger than 1500 bytes on an ethernet subnet, and it also doesn't  
provide any reasons why doing so would be bad. The only thing it says  
is that the MTU option in router advertisements can't be used to  
increase the MTU beyond 1500 bytes.

So in my opinion, an implementation that supports jumboframes should  
use the interface MTU for IPv6 by default, and reduce this MTU for  
IPv6 to the one in an MTU option in router advertisements, when such  
an option is received.

---

From: Ryota Hirose <[email-masked]>
Date: Fri, 22 Jul 2005 10:32:48 +0900 (JST)

Hi,

>From: Iljitsch van Beijnum <[email-masked]>
>Date: Thu, 21 Jul 2005 10:19:56 +0200

> that is apparently incorrect, because it's larger than the standard  

I think so.  When RFC2464 was issued, GbE and Jumbo Frames were not so
deloied, the authors must have believed that the maximum MTU of
ehternet is 1500.  It was reasonable at the time.

> IPv6 to the one in an MTU option in router advertisements, when such  

Yes, your idea is a realistic, and also equal to almost current
implementations, I think.

BTW, suppose following netowrk.

         Internet
            |
          ROUTER
            |
            | 100BASE-TX / MTU=1500
            |
        GbE-SWITCH
         |      |
         |      | 1000BASE-T / MTU=9018
         |      |
       HOST1   HOST2

HOST1, HOST2 and GbE-SWITCH support 1000BASE-T and Jumbo Frames, but
ROUTER has only 100BASE-TX interfaces and doesn't support Jumbo
Frames.  In this network, ROUTER will send RA.  If the MTU option,
which value is 1500, is included in RA, HOST1 and HOST2 will accept
it, and are disabled Jumbo Frames.  So, if we want to use Jumbo
Frambes between HOSTs, (1) HOSTs must neglect MTU option, (2) ROUTER
must send RA without MTU option, (3) or ROUTER must send RA with MTU
option which value is 9018, illegal MTU for ROUTER itself.

(2) RA without MTU option is always valid.  But (1) neglect MTU
option, or (3) MTU option with illegal value for sender itself, are
acceptable?

Ryota Hirose
Yamaha Corporation

---

From: Mark Smith <[email-masked]>
Date: Fri, 22 Jul 2005 14:20:37 +0930

Hi,

On Fri, 22 Jul 2005 10:32:48 +0900 (JST)

> option, or (3) MTU option with illegal value for sender itself, are

I'd think not, probably in either (1) or (3), as an interface MTU value
implies that the other devices attached to the same segment / link can
also receive (and process) the MTU sized frames. 

To support differing MTU values on a single link/segment, each node
would have to somehow keep track of the Maximum Receive Unit value of
its link/segment neighbours. PPP supports doing this, and is able to do
it because a level of layer 2 parameter negotiation goes on before the
link is brought up. It is also simple to do because there is only one
neighbour.

For this sort of thing to be supported on a multi-access media, such as
an ethernet, I'd think some sort of similar, per neighbour, layer 2
parameter negotiation of discovery protocol would have to be developed. 

A solution for your specific scenario would be to have some way of
specifying differing MTUs for onlink and offlink destinations, possibly
in the RAs from the router, such that offlink traffic, which goes via
the router, uses a MTU smaller than or equal to the MRU MTU of the
router's interface. It may be worth developing something like this if
your scenario is common enough. Initially I though it may not be,
however, thinking about it a bit more I could see this set up might be
fairly typical for residential broadband scenarios in the future.

Regards,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Fri, 22 Jul 2005 15:09:42 +0930

On Fri, 22 Jul 2005 14:20:37 +0930

> parameter negotiation of discovery protocol would have to be developed. 

Thinking about this a bit more, this could probably be fairly easy to
achieve by creating a "onlink-MRU" or "interface-MRU" option for ND
Neighbour Advertisements. There'd need to be some logic in how a sending
node selects the size of the packets that it sends to a neighbour, based
on comparing the values of its own local interface MTU, the link's MTU
as announced in the RA(s) and the target neighbours announced
onlink-MRU, if it announces one.

> fairly typical for residential broadband scenarios in the future.

A ND RA option, indicating the "router-MRU" or "offlink-MTU" value might
be necessary and useful also.

Necessary if nodes don't issue ND NSes for routers if they have already
learnt the router's Link layer address via the RA. It may also be useful to
have an onlink-MRU option as well, to allow for local traffic targetted
at the router's onlink interface address(es), which may be used for
administrative access to the router, although probably not necessary for
the amount of onlink traffic a router receives.

Useful if the router only has one other link, e.g., a link to the
Internet, and the MTU of that link is smaller than the onlink MTU, a
good example being a PPPoE 1492 MTU on a ADSL link. The router could
annouce the other link's MTU as the offlink-MTU (assuming it is smaller than the
LAN interface's MTU), which could avoid a PMTUD cycle for every new
offlink connection that encounters the smaller 2nd hop MTU.

If there aren't any big holes in what I'm suggesting, I'm willing to
spend some time co-authoring an Internet draft on this.

Thanks,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Fri, 22 Jul 2005 19:11:13 +0930

On Fri, 22 Jul 2005 15:09:42 +0930

> > > >Date: Thu, 21 Jul 2005 10:19:56 +0200

<snip>

> onlink-MRU, if it announces one.

Looks like this might not be possible, as according to RFC2461,

"     variable MTU   - Neighbor Discovery allows routers to specify a MTU
                      for the link, which all nodes then use.  All nodes
                      on a link must use the same MTU (or Maximum
                      Receive Unit) in order for multicast to work
                      properly.  Otherwise when multicasting a sender,
                      which can not know which nodes will receive the
                      packet, could not determine a minimum packet size
                      all receivers can process."

Certainly understandable, a shame though. I can see more and more SOHO
users buying GigE because it is getting cheap, yet being hobbled to 1.5K
frames because their broadband router doesn't have a GigE interface
supporting jumbo frames. Hopefully SOHO broadband routers start using
GigE jumbo frame capable interfaces, just because they'll be cheap.

Maybe the media default or the RA announced MTU could be used for
multicast traffic, where as a ND discovered best value MRU between
specific neighbours could be used for unicast traffic between them.

<snip>

> offlink connection that encounters the smaller 2nd hop MTU.

If different MTUs were able to be used for multicast and unicast
traffic, then I think the above could also work out.

Regards,
Mark.

---

From: Hugh LaMaster <[email-masked]>
Date: Thu, 21 Jul 2005 12:55:39 -0700

The author of the RFC has clarified (personal communication,
posted with permission) that the intention was never to prohibit
jumbo frames, but rather:

 It is only supposed to mean that Router Advertisements cannot
 increase the MTU above whatever it was otherwise believed to be,
 and that in the absence of configuration information (including
 DHCP), it is presumed to be 1500.

"Please include the admission that perhaps it could have been
clearer.  Remember the old "We'll fix it in Draft" mantra that
used to be the ipngwg refrain?"

---

From: "Stephen Sprunk" <[email-masked]>
Date: Fri, 22 Jul 2005 13:20:40 -0500

Thus spake "Mark Smith" <[email-masked]>
> Neighbour Advertisements. There'd need to be some logic in how a
> sending node selects the size of the packets that it sends to a
> neighbour, based on comparing the values of its own local interface
> MTU, the link's MTU as announced in the RA(s) and the target
> neighbours announced onlink-MRU, if it announces one.
...
> spend some time co-authoring an Internet draft on this.

The hole is that there may be an L2 device in the middle which has a lower 
MTU than the end hosts.  The neighbor MTU is an upper bound, but it's not 
guaranteed to work -- you need to probe to see what really works.

Just like PMTUD, you need to periodically probe and adjust to changing 
network conditions, including detecting "black holes".  Fred Baker suggested 
the host send both minimum MTU (576 for v4 and 1280 for v6) and maximum MTU 
frames in a given burst and track what gets through.  Obviously if the 
larger frames work you go with that, but if not you need to ratchet the max 
MTU down until things work (while using the min MTU so the session doesn't 
stall).

The most perverse scenario I can envision is a network where one host has an 
MTU of 9k, another has 8k, one network path has 10k, another path has 3k, 
and the path varies every few minutes (and isn't necessarily symmetric). 
Obviously it needs to be stated that MTU changes need to be communicated up 
to TCP for MSS adjustments -- darn layer violations.

One thing we probably need to decide is if it's "correct" for the above 
hosts to (a) fall back to 1500, (b) use 3k all the time, or (c) alternate 
between 3k and 8k.  I'm leaning towards all three being legal with a 
recommendation for the second option

S

Stephen Sprunk      "Those people who think they know everything
CCIE #3723         are a great annoyance to those of us who do."
K5SSS                                             --Isaac Asimov

---

From: Fred Baker <[email-masked]>
Date: Fri, 22 Jul 2005 12:03:48 -0700

Actually, I should think 1280 would be fine for both. The issue tends  
to be IPSEC tunnel mode and other tunneling implementations in the  
path in between, otherwise I would say "ip data size == 1500". 576  
byte frames are a leftover from ARPANET IMPs and Fuzzballs...

The suggestion actually comes from Matt's document, I just pointed it  
out and suggested a simple first-burst implementation.
   http://www.ietf.org/internet-drafts/draft-ietf-pmtud-method-04.txt
   "Path MTU Discovery", Matt Mathis, 22-Feb-05

In context, Max MTU is the max on the local interface. If you're on a  
10/100 Ethernet and you're in an end station, you probably don't do  
9K byte MTUs, while on a GigE interface you probably do. Depending on  
the implementation, you might try to send an 8192 byte ip datagram no  
matter what and it might not get out of your system, or you may be  
able to read the MTU size.

On Jul 22, 2005, at 11:20 AM, Stephen Sprunk wrote:

> Just like PMTUD, you need to periodically probe and adjust to  
> changing network conditions, including detecting "black holes".   
> Fred Baker suggested the host send both minimum MTU (576 for v4 and  
> 1280 for v6) and maximum MTU frames in a given burst and track what  
> gets through.  Obviously if the larger frames work you go with  
> that, but if not you need to ratchet the max MTU down until things  
> work (while using the min MTU so the session doesn't stall).
>

---

From: Bob Hinden <[email-masked]>
Date: Fri, 22 Jul 2005 13:18:24 -0700

Hi,

>Frames.

In my personal view, having devices on shared media with different MTU's is 
just a bad idea.   Trying to get it to work is complicated and I think it 
will have lots of nasty failure cases.  If one wants to have mixed speeds 
(like the example above) then use the default MTU (i.e., 1500) or replace 
the GbE-Switch with a Router.  If one wants Jumbo Frames, then make sure 
all of the nodes on the link support 1000Base-T.

Bob

---

From: Fred Baker <[email-masked]>
Date: Fri, 22 Jul 2005 14:00:40 -0700

In my personal view, networks like this are a fact of life. One  
frequently is handed things by disparate people with disparate  
budgets, responsibilities, and requirements. For example, perhaps IT  
provides the switch, the router is what the managed-service ISP  
installed several years ago and still works, and the servers are what  
the department bought for this nifty-keeno application they're in the  
process of installing.

Hence, path MTU is a question of what's on the path for a session,  
not something else.

On Jul 22, 2005, at 1:18 PM, Bob Hinden wrote:

> In my personal view, having devices on shared media with different  
> MTU's is just a bad idea.   Trying to get it to work is complicated  
> and I think it will have lots of nasty failure cases.  If one wants  
> to have mixed speeds (like the example above) then use the default  
> MTU (i.e., 1500) or replace the GbE-Switch with a Router.  If one  
> wants Jumbo Frames, then make sure all of the nodes on the link  
> support 1000Base-T.
>
> Bob
>
>
>
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 23 Jul 2005 02:50:58 +0200

On 22-jul-2005, at 20:20, Stephen Sprunk wrote:

>> Neighbour Advertisements.

>> spend some time co-authoring an Internet draft on this.

I'm sure we can find a nice restaurant or café in Paris to discuss  
the matter further.  :-)

> The hole is that there may be an L2 device in the middle which has  
> a lower MTU than the end hosts.  The neighbor MTU is an upper  
> bound, but it's not guaranteed to work -- you need to probe to see  
> what really works.

(Layer 1 devices can also impose MTU limits.)

It's important to keep this simple, unless the IEEE guys also want to  
play and add mechanisms to exchange MTU information between switches.

> gets through.

I'm not really comfortable with this... It makes more sense to me to  
have a router or two, or maybe one or two non-router hosts, send out  
"MTU announcements", and other hosts only announce the non-standard  
MTU in neighbor advertisements when they recently heard one of those  
announcements. When the MTU suddenly decreasees, the announcements  
are no longer heard, hosts put 1500 in their neighbor advertisements  
and neighbor unreachability detection does the rest. The fact that  
ethernet is supposed to have a tree topology makes things slightly  
simpler.

> The most perverse scenario I can envision is a network where one  
> host has an MTU of 9k, another has 8k, one network path has 10k,  
> another path has 3k, and the path varies every few minutes (and  
> isn't necessarily symmetric).

Real-life ethernet isn't supposed to be like that...

For those who think there isn't a real problem here: it takes a  
little over 800 packets per second to saturate a 10 Mbps ethernet  
link. At GE speeds that's 80000 packets per second. It is very hard  
to achieve decent performance when you have to stop what you're doing  
80000 times per second... There is also the environment to consider  
because the amount of power switches use is strongly related to the  
number of packets that flow through the switch. So increasing the MTU  
from (for instance) 1500 to 9000 bytes means it only takes 3 packets  
to transfer 18000 bytes (2 data, 1 ack), while it takes (best case)  
13 packets at 1500 bytes (12 data, 1 ack) but usually 18 (6 acks).  
That saves a LOT of power.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 23 Jul 2005 02:57:09 +0200

> wants Jumbo Frames, then make sure all of the nodes on the link  

Unfortunately 1000baseT (is that the correct designation?) doesn't  
equal jumboframe capability, so the problem remains.

Don't forget that management issues in layer 2 networks (= a single  
layer 3 subnet) are very different from those on the global internet:  
most layer 2 networks are managed by a single organization. Since we  
had the hubris to think we could do PMTUD for the global internet, I  
think we shouldn't shy away from doing the same thing for layer 2. As  
always, the operator community will decide wheter what we come up  
with is usable in practice.

---

From: Perry Lorier <[email-masked]>
Date: Sun, 24 Jul 2005 03:12:34 +1200

>> The hole is that there may be an L2 device in the middle which has  a
>> lower MTU than the end hosts.  The neighbor MTU is an upper  bound,
>> but it's not guaranteed to work -- you need to probe to see  what
>> really works.
> 
> 

Before two hosts can transmit packets on Ethernet they have to undergo
neighbour solicitation to find the remote ends hardware address anyway.

When you send the neighbour solicitation you could pad the packet out
with multiple "MTU padding" options to make the packet the size of the
MTU you want to use.

The remote end receives the message, and adds it's own padding[1] to the
reply.

If the original host receives a message from the remote host with MTU
padding options then it knows the remote host can receive a message at
least that big and uses that as the remote hosts MTU.

If the original host does not receive a reply, it updates it's neighbour
cache to say that the MTU supported is the MTU announced by the RA's, if
no MTU is announced then it uses a configured minimum MTU for that link
 and retransmits the query without any "MTU padding" options.

This supports multiple MTU's on the same link.  This consumes more
bandwidth (the padding) during neighbour solicitation which is a "rare"
event (only once per host pair, not per connection, or worse, per
packet).  Neighbour solicitation messages should go to multicast
addresses that are likely to be used by at most only a few nodes, so the
extra bandwidth used by the padding isn't excessive.  This doesn't
introduce more round trips however step down could be implemented with
more round trips if desired.  The test is "admin proof", admins
shouldn't need to do any configuration for a host to start using large
MTU's. If the network is poorly designed with an "invisible" low MTU
link then the hosts will fall back correctly.  This should recover from
changes in MTU since new neighbour discoverys will discover the new link
MTU.  It's backwards compatible (nodes that don't understand this
protocol MUST silently ignore the option according to RFC2461).  The
test should be reasonably robust, if a network cannot support[2] large
mtus then the protocol will fall back to the minimum size.

Problems with this protocol include that it can detect the existence of
a "jumbogram path", and if it doesn't find it at the "preferred MTU"
then it falls back to the lowest common denominator and doesn't detect
intermediate sizes.  It "wastes" bandwidth by padding neighbour
solicitation/advertisement packets.  This also assumes that MTU paths
are symmetric.

Options can only be 256 bytes long (there is an 8 byte length field for
options), so a lot of options are required to pad a neighbour
solicitation/advertisement to 9k.  This could be resolved by changing
the neighbour discovery protocol to use another (longer) length field if
the length field is 0 (RFC 2461 explicitly says you must drop packets
that have this set).  This change would have much more impact however.

If this sounds insane, in my defense it's 3am and sounded like a good
idea at the time :)

Perry

----
[1]: A host for instance may support receiving 9k MTU packets, but that
doesn't mean that it doesn't prefer them to be under 4k, so a host can
ask for it to be sent 4k MTU packets.

[2]: Cannot support because of MTU issues, or due to high loss rates etc.

---

From: Mark Smith <[email-masked]>
Date: Sun, 24 Jul 2005 01:38:55 +0930

On Fri, 22 Jul 2005 14:00:40 -0700

> the department bought for this nifty-keeno application they're in the  

I see above scenario possibly becoming quite common in the near future
in a residential/SOHO scenario.

GigE NICs and switches are dramatically coming down in price, and are
commonly available in "commodity" type computer and office shops, where
"joe six pack" also goes to buy his PC supplies.

While the low end GigE switches don't seem to support jumbo frames yet,
I doubt it will be long before they do. When jumbo frame capable
commodity switches become available, their owners are unlikely to also
be prepared to throw away their still quite capable ADSL/Cable router,
yet will be disappointed when they aren't able to take advantage of the
jumbo frame performance they've just spent money on between the nodes
they've just installed GigE NICs in.

Regards,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Sun, 24 Jul 2005 01:56:54 +0930

Hi Stephen,

On Fri, 22 Jul 2005 13:20:40 -0500

> guaranteed to work -- you need to probe to see what really works.

I agree that probing would be nice to have, however I don't think not
having it is a "black" hole.

When a larger than standard MTU is used (e.g., 9K jumbos over GigE
verses the 802.3 standard of 1500), the operator has made a conscious
decision to increase the MTU, and therefore needs to take responsibility
for ensuring that intermediary layer 2 devices support the increased
MTU. People who want use jumbo frames on GigE interfaces know usually
that they have to also have jumbo frame capable switches..

If we make the assumption that the intermediary link layer devices support the
largest node MTU/MRU on the segment, I think the problem becomes a lot
simpler, and then the issues to solve are :

(a) how to ensure large MTU/MRU capable end nodes use them when
communicating between each other.

(b) how to ensure end-nodes with smaller MTU/MRUs are not sent too large
frames by the large MTU/MRU capable end-nodes.

I think either of these issues could be solved by using a ND NS/NA MRU
type option, and, because RAs can carry a link-layer address, negating
the need for a node to perform a ND NS/NA transaction with the router,
at least initially, have an RA also possibly carry an MRU indication.

For multicast, the link standard MTU or the link RA announced MTU would
be used.

For nodes that don't implement this "MRU" option, they'll use the link
standard MTU to communicate with their neighbours and vice-versa, as per
IPv6 operation today.

I do agree this is in some respects a half-way step - it would be better
if a method of probing could take place within the link to cope with a
diversity of MTU/MRU capabilities and limitations, including those of
intermediarly layer 2 devices. Still, I think if the above allows people
to mix and match differing speed and MTU nodes on a layer 2
infrastructure that is known to support the largest-possible node
MTU/MRU, it could be beneficial.

Regards,
Mark.

---

From: "Joseph T. Klein" <[email-masked]>
Date: Sat, 23 Jul 2005 12:47:41 -0500

Perhaps this wheel has been invented before? This sounds like MTU
discovery. RFC 1063 ??

---

From: "Stephen Sprunk" <[email-masked]>
Date: Sat, 23 Jul 2005 14:45:46 -0500

Thus spake "Iljitsch van Beijnum" <[email-masked]>
> I'm sure we can find a nice restaurant or café in Paris to discuss  the 
> matter further.  :-)

I'm not able to attend, so I'd appreciate it if someone would give my 
comments a bit of airtime even if none of the attendees agree with me.

>> lower MTU than the end hosts.  The neighbor MTU is an upper  bound, but 
>> it's not guaranteed to work -- you need to probe to see  what really 
>> works.
>

True; read the above L2 as L1/L2.

> It's important to keep this simple, unless the IEEE guys also want
> to play and add mechanisms to exchange MTU information between
> switches.

Given their past stance on jumbos, I don't see that happening.

>> network conditions, including detecting "black holes".   Fred Baker 
>> suggested the host send both minimum MTU (576 for
>> v4 and 1280 for v6) and maximum MTU frames in a given burst
>> and track what gets through.
>
> I'm not really comfortable with this... It makes more sense to me to  have 
> a router or two, or maybe one or two non-router hosts, send out  "MTU 
> announcements", and other hosts only announce the non-
> standard MTU in neighbor advertisements when they recently heard
> one of those announcements. When the MTU suddenly decreasees,
> the announcements are no longer heard, hosts put 1500 in their
> neighbor advertisements and neighbor unreachability detection does
> the rest.

I'm okay with hosts dropping to 1500 (assuming Ethernet) for neighbors that 
can't receive jumbos.  It'd be desirable for them to find a higher value 
that works but which is still less than both hosts' MTUs.  Trying to ratchet 
the MTU back up after it's been lowered is probably more trouble than it's 
worth.

> The fact that ethernet is supposed to have a tree topology makes
> things slightly simpler.

Some 802 networks are not trees, and there are non-802 networks out there. 
I'd like to have a single jumbo spec for all L1/L2 types; it doesn't seem to 
require much tap-dancing around the specific numbers to generalize it to all 
media types, though obviously Ethernet is the most common and most in need 
of help.

>> The most perverse scenario I can envision is a network where one  host 
>> has an MTU of 9k, another has 8k, one network path has 10k,  another path 
>> has 3k, and the path varies every few minutes (and  isn't necessarily 
>> symmetric).
>

I've been traumatized by some of the networks I've seen in the wild.  The 
scenario I gave was deliberately perverse, but it's not very far from the 
worst I've encountered -- and I guarantee such things will occur if we 
standardize jumbos.  I'm betting that's why the IEEE refused to tackle it.

> For those who think there isn't a real problem here: it takes a  little
> over 800 packets per second to saturate a 10 Mbps ethernet link.
> At GE speeds that's 80000 packets per second. It is very hard to
> achieve decent performance when you have to stop what you're doing  80000 
> times per second...

Modern hosts can do it, but it'd be nice to reduce the CPU load due to NIC 
interrupts if possible.

> There is also the environment to consider because the amount of
> power switches use is strongly related to the number of packets that
> flow through the switch. So increasing the MTU from (for instance)
> 1500 to 9000 bytes means it only takes 3 packets to transfer 18000
> bytes (2 data, 1 ack), while it takes (best case) 13 packets at 1500
> bytes (12 data, 1 ack) but usually 18 (6 acks).  That saves a LOT
> of power.

The power consideration didn't even occur to me; I was thinking of CPU load 
on end hosts, per-packet overhead on the wire, and pps limits on network 
gear.

S

---

From: "Stephen Sprunk" <[email-masked]>
Date: Sat, 23 Jul 2005 14:16:46 -0500

> When a larger than standard MTU is used (e.g., 9K jumbos over
> GigE verses the 802.3 standard of 1500), the operator has made a
> conscious decision to increase the MTU, and therefore needs to
> take responsibility for ensuring that intermediary layer 2 devices
> support the increased MTU. People who want use jumbo frames
> on GigE interfaces know usually that they have to also have
> jumbo frame capable switches..

Except we nearly have this situation today -- jumbos only work if the admin 
goes through a lot of painstaking effort to make sure they do.  You're 
proposing we eliminate the per-host effort but not the network-side effort; 
to me that's a half-solution (as you granted in a section I snipped).  I 
want jumbos to work out of the box without the admin (which might be my 
grandmother at home) even knowing what jumbos are.

> If we make the assumption that the intermediary link layer devices
> support the largest node MTU/MRU on the segment, I think the
> problem becomes a lot simpler, and then the issues to solve are :
>
> (a) how to ensure large MTU/MRU capable end nodes use them
> when communicating between each other.
>
> (b) how to ensure end-nodes with smaller MTU/MRUs are not sent
> too large frames by the large MTU/MRU capable end-nodes.
>
> I think either of these issues could be solved by using a ND NS/NA
> MRU type option, and, because RAs can carry a link-layer address,
> negating the need for a node to perform a ND NS/NA transaction
> with the router, at least initially, have an RA also possibly carry
> an MRU indication.

The RA's MRU would make a good upper bound, but you still need to probe to 
make sure a host you're talking to isn't sitting behind some $30 hub that 
silently drops jumbos (er, giants).  And you have to keep probing in case 
the host moves or the L1/L2 path changes, unless you've recently received 
from that host a jumbo or ACK for your own jumbo.

The reason I think this is necessary is that it allows hosts supporting 
jumbos to have them on by default and gracefully fall back to non-jumbo 
sizes in the presence of non-jumbo-aware hosts or network devices.  You can 
only have them on by default if you're sure you can handle the most perverse 
case (which I think I presented).

> For multicast, the link standard MTU or the link RA announced MTU
> would be used.

I'm not sure using an MTU above 1500 for multicast should be legal; there 
may be hosts on the subnet which we don't know are neighbors and thus we 
don't know the lowest MRU out there.  Also, aren't there cases where we'll 
know of a neighbor but wouldn't have had to do ND and thus wouldn't have 
learned their MRU?

> standard MTU to communicate with their neighbours and vice-versa,
> as per IPv6 operation today.

Of course.

And, since I know this is the IPv6 WG list, which WG would be appropriate to 
discuss back-porting this feature into IPv4 after we solve it for IPv6?

S

---

From: Mark Smith <[email-masked]>
Date: Sun, 24 Jul 2005 12:21:28 +0930

On Sat, 23 Jul 2005 14:16:46 -0500

> grandmother at home) even knowing what jumbos are.

Remember that the per-host effort is where a lot of the effort is -
there are just so many more of them than there are layer 2
infrastructure devices.

I'd like "ultimate plug-and-play" for this scenario, however, for quite
valid reasons, that problem is either quite hard or even impossible to
solve. 

The fundamentaly issue I think I'm really addressing is that even if the
administrator goes to the effort of ensuring that the layer 2
intermediary devices support jumbo frames, IPv6 currently prevents you
taking advantage of it for the hosts that support it, the moment you
plug in one legacy device that doesn't.

For example, you may have a few hundred hosts that support jumbo frames,
and layer 2 switching infrastructure that also supports jumbo frames.
You then want to plug in an older 10 or 100Mbps network attach printer
(that has been software upgraded to support IPv6) that doesn't. That
printer hobles all the hosts to using 1500 byte frames for IPv6. Sure,
you can fix that by buying or building a 2 interface router that
supports jumbo ethernet frames on at least one interface, and then
plugging the printer into the other interface, creating a different
MTU'd segment. Thats a fairly $$$$ or time intensive solution. I'd think
modifying IPv6 to support this scenario (again, assuming layer 2
infrastructure supports the largest MTU/MRU) would be very much
appreciated by people who design a network like this and are then faced
with the idea of having to buy/build a router just to support an
existing, legacy printer.

"Legacy" printers could with any other "legacy hardware" device that has
been software upgraded to support IPv6, e.g. router, host, scientific
equipment etc.

> from that host a jumbo or ACK for your own jumbo.

If you've gone to the effort of consciously building a jumbo capable
infrastructure, and then chosen to use it, and then somebody else
('cause it won't be you) plugs in a $30 hub, then 

(a) you can repremand them

(b) be sure that the problem will be localised to the devices on the
other side of the hub, if they try to plug jumbo capable devices in
behind it, and attempt to use jumbo frames. The existing, properly
supported jumbo capable devices plugged into the proper jumbo capable
layer 2 infrastructure would continue to work properly. If they plug in
jumbo frame capable devices behind the hub (forgetting for the moment
that hubs don't come in GigE flavour), but don't enable jumbo frames on
the end-nodes, things will work as normal.

Again, remember that my suggestion solution is _constrained_ to
scenarios where the layer 2 infrastructure has been specifically
designed to support jumbo frames, and a conscious effort has been made
to enable them. My solution has the fundamental goal of allowing
small-MTU, "legacy" end-nodes (e.g., printers, routers, etc.) to be
plugged into the same layer 2 infrastructure without those legacy
devices hobbling the MTU/MRUs the jumbo capable devices support.

I think this scenario is or will be quite common in the future. I'd
suggest that buying GigE layer 2 switching without looking at its
features, capabilities and specifications, is only limited to the
residential, SOHO and maybe small business market. IOW, any scenario
where the network is actually designed may benefit from this MTU/MRU
discovery solution for unicast traffic. Those networks are present in
medium to large businesses, and service providers, and I think there
will be lots of "_designed_ to support jumbo" frame networks in the
future.

> know of a neighbor but wouldn't have had to do ND and thus wouldn't have 

They wouldn't be legal, as the multicast scenario is exactly why RFC2461
says the MTU has to be of a fixed size for the segment. However, once we
apply this constraint only to multicast traffic, we then may be able to
allowing differing MTU/MRUs for unicast traffic between hosts on the
same link (of course, again, assuming that the layer 2 infrastructure
supports the largest MTU/MRU).

> discuss back-porting this feature into IPv4 after we solve it for IPv6?

We choose not to back-port (it may not be possible anyway as I don't
think ARP is extensible like IPv6 ND is), and that will then provide
another reason to migrate to IPv6 ... :-)

Back-porting nice features from IPv6 to IPv4 is possibly false economy.
It takes extra effort (even though it may be a small amount), which
instead could be used to further improve IPv6, and also provides
additional disincentives to adopt IPv6. In some ways it reminds me of
the effort that is spent overcoming NAT limitations, rather than more
usefully either improving the application by adding features, or porting
the application to IPv6 and avoiding then avoiding NAT issues
completely.

Regards,
Mark.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 24 Jul 2005 13:40:53 +0200

> neighbour solicitation to find the remote ends hardware address  

Right.

> with multiple "MTU padding" options to make the packet the size of the

I don't think we want to do this in the neighbor _solicitation_  
because there is a good chance that the neighbor doesn't support the  
larger MTU so the packet is lost.

>  and retransmits the query without any "MTU padding" options.

This is problematic for two reasons: when the packet gets lost  
because either the receiver or the layer 2 network don't support a  
sufficiently large MTU/MRU, there is a timeout, which wastes time,  
and if the receiver does in fact support jumboframes but of a smaller  
size than the sender supports, this isn't detected.

> If this sounds insane, in my defense it's 3am and sounded like a good

:-)

I think our requirements are:

1. do not impede 1500-byte operation
2. discover and utilize jumboframe capability where possible
3. discover and utilize (close to) the maximum MTU
4. recover from sudden MTU reductions fast enough for TCP and similar  
to survive

First of all, we need for hosts to find out that their correspondents  
support a larger MTU/MRU. This can easily be done in an ND option.

Since we're not going to get cooperation from switches, let alone  
hubs, it's important that we send test packets to see whether the  
jumboframes actually make it to the other side. I think using ND for  
this isn't a good idea: in its current form, it doesn't support the  
required packet size, and when the padded packet doesn't make it,  
there recovery complexities and delays.

So it makes sense to come up with a new protocol for this. An  
interesting notion here is that this protocol doesn't have to be IPv6- 
specific. However, in IPv6 we have neighbor unreachability detection  
which we can use to find MTU reductions fast enough to fall back to  
1500 bytes before bad things happen. In IPv4 or pure ethernet, we  
don't have that, and we also don't have neighbor discovery to  
exchange per-host MRU/MTU information.

If we use IPv6 for this, I think a new ICMP type makes sense.  
Whenever two systems (hosts or routers) on a link perform neighbor  
discovery, they can trigger the MTU verficiation immediately  
afterward, and if jumboframe support is confirmed by receiving the  
larger packets, the MTU for the the neighbor can be updated. If the  
larger packets don't make it to the neighbor there is no complexity  
and no delay: communication was already underway at 1500 bytes and  
continues without the need for further action.

However, this doesn't accommodate finding out jumboframe support at  
reduced sizes very well. For this, I think we should use an  
additional exchange, but this one should probably happen over  
multicast. Hosts/routers could take turns in a distributed search for  
the largest supported framesize. I think it's important that all  
jumbo-capable systems take part in this in order to deal with unusual  
topologies. For instance, consider a network with three switches: one  
support 9000, another 8000 and the two are connected through a third  
switch that only supports 3000 bytes:

   A                C
   |                |
+-+--+  +----+  +--+-+
|9000+--+3000+--+8000|
+-+--+  +----+  +--+-+
   |                |
   B                D

Suppose all hosts support 9216 byte jumboframes.

I think the most efficient way to handle this is to do two concurrent  
searches: one for the maximum packet size that can be used to at  
least one correspondent, and one for the minimum jumboframe size that  
is supported by all jumboframe supporting systems.

So first A sends out an announcement that it's going to send a 9216  
byte and a 5596 (1500 + 4096) byte packet, and then sends the  
packets. Nobody receives the first packet, but everyone knows A sent  
it because of the preceding announcement, and B receives the second  
packet.

Then B would (for instance) send out its 9216 byte packet along with  
a 1500 + 2048 = 3548 byte packet, and also indicates the largest size  
that worked (5596) and the smallest size that didn't work (9126). A  
receives the 3548 byte packet but not the 9216 byte one.

C is next and sends out 9216 and (1500 + 1024 = ) 2524 byte packets,  
along with the information that no jumboframe size has worked so far.  
A, B and D all receive the 2524 byte packet.

D then sends out 9216 and (1500 + 1536 = ) 3036 byte packets with  
information that it received 2524 but not 3548. C receives the 3036  
byte packet.

It's now A's turn again. A knows that the size that everyone can  
receive is betweeen 2524 and 3036 and the size that at least one  
correspondent can receive is between 5596 and 9216. So it sends out  
2780 and 7406 byte packets.

And so on.

After a few round like this, each system knows the maximum jumboframe  
size it can send/receive (so it can adjust its announcements in the  
ND option), and the minimum jumboframe size that everyone supports.  
It's probably doable to generalize this into any given number of  
levels, but I doubt that more than 3 is worth the trouble, and maybe  
having two levels even isn't. On the other hand, if some hosts  
support 9000 but the majority support 8192 it may be a good idea to  
forget 9000 and just do 8192.

This may sound horribly complex, but it really isn't.  :-)

The biggest challenge is probably making the different systems talk  
in turn, but that can probably be done by having a timer that depends  
on the difference in MAC address between the last system to transmit  
and prospective next one.

Extra credit: monitor spanning tree events for quick adaption to  
changing layer 2 topologies.

Alternatively, we could add an RA option that administrators can use  
to tell hosts the jumboframe size the layer 2 network supports. (The  
RA option doesn't say anything about the capabilities of the  
_router_.) Then the whole multicast taking turns discovery isn't  
necessary, and we can suffice with a quick one-to-one verification  
before jumboframes are used.

---

From: Perry Lorier <[email-masked]>
Date: Mon, 25 Jul 2005 01:16:34 +1200

<I agree with pretty much everything said here, so I'll just snip it>

> 4. recover from sudden MTU reductions fast enough for TCP and similar

I'd add to this list:
5. Must be fully automatic and not require any admin intervention to do
the "right" thing.
6. Minimise the resources used.

> 1500 bytes before bad things happen. In IPv4 or pure ethernet, we  don't
> have that, and we also don't have neighbor discovery to  exchange
> per-host MRU/MTU information.
> 

Yep, makes sense to me. :)

> continues without the need for further action.

Yep!  I'd considered this, but at 3am didn't want to confuse the issue
by introducing too many differing ideas at the same time. (Also, I
wanted to go to sleep :)  You've seemed to have thought through some of
the issues a bit better than I had too :)

> reduced sizes very well. For this, I think we should use an  additional
> exchange, but this one should probably happen over  multicast.

I disagree.  There is no need to for every host to have a full
understanding of the layer 2 topology of the network it is on.  We're
starting to see some very large L2 networks as MAN's (eg NLR[1]) and
IPv6's /64 per subnet puts no real practical limit on how large a single
L2 segment can be.  The storage requirements for all of this information
alone is either going to be very large (remembering MTU's for every
neighbour on the link?!) and/or it's going to have be repeated almost
constantly as hosts want to discover MTU's to specific neighbours.

Nodes that are reasonably "idle" on the link are going to have to
participate in this and therefore avoid entering low power modes.

Multicasting out "jumbo" packets is going to tax switch queue sizes to
the extreme.  A 48 port switch, receiving a single multicast 9k packet
(that it can forward) may end up with 48*9k=430,000 bytes (or nearly 1/2
a megabyte) of buffer space used with a single packet arriving.

What happens on l2's where not every node can see every other node?
Some L2's only allow end hosts to talk to a master host.  What about a
network with vlans where some hosts are on differing sets of vlans?

> Hosts/routers could take turns in a distributed search for  the largest
> supported framesize. I think it's important that all  jumbo-capable
> systems take part in this in order to deal with unusual  topologies. For
> instance, consider a network with three switches: one  support 9000,
> another 8000 and the two are connected through a third  switch that only
> supports 3000 bytes:
> 
> 
>   A                C
>   |                |
> +-+--+  +----+  +--+-+
> |9000+--+3000+--+8000|
> +-+--+  +----+  +--+-+
>   |                |
>   B                D
> 

If A and B are talking to each other and C and D are talking to each
other, why do (A and B) need to talk to C and D?

> searches: one for the maximum packet size that can be used to at  least
> one correspondent, and one for the minimum jumboframe size that  is
> supported by all jumboframe supporting systems.

Why not a simpler protocol?

Host A sends a ND to Host B with Host A's MRU.
Host B replies with a NS to Host A with Host B's MRU.
Host A can now start transmitting the data it wants to send.
Host A now sends Host B a ICMP MTU Probe, at some size less than or
equal to Host B's MRU
If Host B receives the packet, it replies with an ICMP MTU Probe reply
saying what it received.

By working "up" known sizes instead of doing a binary search (or working
down), Host A can quickly ratchet up sizes without waiting for a timeout
gaining immediate benefits from larger MTUs as they are discovered.
Which sizes should it use?  Well, there probably should be a hard coded
list of "well known MTU sizes" which could perhaps be expanded at
runtime (if you find an host with MTU that's not in that list, then add
the MTU to it so you can more quickly find that MTU in future).

> byte and a 5596 (1500 + 4096) byte packet, and then sends the  packets.
> Nobody receives the first packet, but everyone knows A sent  it because
> of the preceding announcement, and B receives the second  packet.

If you persist with this idea, switch the order of the packets sent.
Send the packet first, then send the announcement that it was sent.
Assuming no reordering you then don't have to wait for a timeout.  If
reordering does occur you then send a "Whoops! reordering! didn't expect
that on the same L2!" and then everyone flags that interface as
"possible reordering" and then always waits for a timeout.

In the common case of no reordering this will be much faster due to not
waiting for timeouts.

> Then B would (for instance) send out its 9216 byte packet along with  a
> 1500 + 2048 = 3548 byte packet, and also indicates the largest size 
> size it can send/receive (so it can adjust its announcements in the  ND
> option), and the minimum jumboframe size that everyone supports.  It's
> probably doable to generalize this into any given number of  levels, but
> I doubt that more than 3 is worth the trouble, and maybe  having two
> levels even isn't. On the other hand, if some hosts  support 9000 but
> the majority support 8192 it may be a good idea to  forget 9000 and just
> do 8192.
> 
> The biggest challenge is probably making the different systems talk  in
> turn, but that can probably be done by having a timer that depends  on
> the difference in MAC address between the last system to transmit  and
> prospective next one.

Not everything has a MAC address.  Difference in link local addresses?
This sounds very much like turning Ethernet into token ring <grin>.

> changing layer 2 topologies.

Ooh, very good call.  I'd not considered this!

> Alternatively, we could add an RA option that administrators can use  to
> tell hosts the jumboframe size the layer 2 network supports. (The  RA
> option doesn't say anything about the capabilities of the  _router_.)
> Then the whole multicast taking turns discovery isn't  necessary, and we
> can suffice with a quick one-to-one verification  before jumboframes are
> used.

This still seems to fall foul of either requiring the administrator to
configure the router or degrading the entire network to the level of the
router.  Since routers are often the bottleneck (eg DSL in NZ is
currently <=~2mbit, so why have a DSL router thats faster than 10mbit?
10mbit still has a low MTU, but hosts within the site should still try
and use jumbograms)

----
[1]: National Lambda Rail claim to be the first national switched
Ethernet (http://www.nlr.net/architecture.html)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 24 Jul 2005 20:06:22 +0200

> 6. Minimise the resources used.

Agree, except that packets are cheap on a 1000 Mbps LAN, so those  
don't count much towards 6.

(-:

>> additional

> understanding of the layer 2 topology of the network it is on.

That's not what the mechanism that I outlined does. What it does do  
is let all jumbo-capable system send send at least one packet at  
their maximum size and get feedback on whether it was received by  
anyone. For that, every packet may update information that the system  
(host or router) holds, but then the old information is gone so there  
is no per-(potential) correspondent state.

> We're
> L2 segment can be.

Hm, they invented routers for a reason, I don't think we have to bend  
over backwards to accommodate unreasonably large layer 2 networks.  
But: what would be reasonable to accommodate? 1000 systems?

> a megabyte) of buffer space used with a single packet arriving.

I'm not sure this is a problem: as far as I know (and that's not too  
far) switches use per-port buffer space, so although such a packet  
uses up buffer space for a lot of ports, there is nothing  
unreasonable about that (packet is gone again in 72 microseconds  
anyway). But some vendor feedback would be good here.

> What happens on l2's where not every node can see every other node?

Neighbor discovery fails?

> network with vlans where some hosts are on differing sets of vlans?

I don't think those exist in that way.

> other, why do (A and B) need to talk to C and D?

Ah, but how do you know that A doesn't talk to D, and is never going to?

> If Host B receives the packet, it replies with an ICMP MTU Probe reply

The problem here is that if the switches in the middle don't support  
jumboframe sizes A and B do, you can't use jumboframes at all. Since  
there is no standard jumboframe size, this is a problem. Of course A  
and B can discover the maximum size between them, but doing that  
between any set of correspondents seems suboptimal.

On the other hand, if we reuse the information learned for previous  
correspondents, this could work well.

So:

When ND indicates a neighbor supports jumboframes, we start by  
sending an MTU prope with either an MTU that worked towards another  
correspondent fairly recently (last hour or so). If the correspondent  
receives this packet it sends an ack and both sides can increase the  
MTU.

However, it's possible we can improve on this MTU, so now we do a  
binary search between the current MTU and the maximum usable one (=  
minimum of local and remote MTU/MRUs), at one packet per second or so.

If the first probe with a recently used MTU fails then we do a binary  
search betwen that value and an initial one, which should probably be  
1508 (some NICs don't do jumboframes but support 1504 for VLAN use,  
nearly all MTUs are 32 bit aligned, the maximum 3 extra bytes aren't  
worth it anyway).

I think we can assume the MTU is the same in both directions. So when  
system A tries with 3000 bytes (worked with C!) towards B, B sets an  
ack flag and tries with 9216, which fails, so A sends a NAK and tries  
with 6108, and so on. With each successful packet (either received or  
acknowledged) the MTU towards the correspondent can be increased  
immediately.

> gaining immediate benefits from larger MTUs as they are discovered.

You can do this with binary search too, as long as you send a  
separate "now testing YYYY, ack XXXX" packet.

I don't think we want to do this at top speed, though, because the  
control traffic could get in the way of real traffic.

> Send the packet first, then send the announcement that it was sent.

Yes, that makes sense. I thought there would be a possibility that  
the switch or hub would need some time to recover after receiving a  
packet that's too large, though.

> In the common case of no reordering this will be much faster due to  

If the "I sent..." packet comes in before the actual test packet,  
then this would look like the test packet didn't make it, so the  
receiver would send a NAK. However, if the packet does make it and  
comes in late, then obviously the receiver notices this and it can  
send out an ACK as well. Since we don't want to hammer the layer 2  
network with possibly invalid packets, the receiver (well, sender of  
the original probe) would probably want to wait long enough for that  
ACK to come in before sending the next packet anyway.

> Not everything has a MAC address.

Yikes! You really are a modern day René Descartes, aren't you?  :-)

> Difference in link local addresses?

Ring networks are very cool, too bad we don't have them anymore.

> configure the router

Well, that's what administrators do, isn't it?

> or degrading the entire network to the level of the
> router.

No, the announcement "the switch can handle 4500 bytes" wouldn't have  
anything to do with "I can handle 1500".

It's probably a good idea to make announcements like this part of the  
protocol, but not as RA options. That way, switches can announce  
their own MTU capabilities, even if they don't otherwise support  
IPv6. So if the switch says that it can do 4500, we only have to try  
4500 (ack) and 4504 (nak) and everything is much faster. (Unless the  
layer 2 network is more complex, of course, but then either 4500 gets  
a nack/timeout or 4504 gets an ack.)

Hm, maybe 4 bytes larger than an earlier maximum is always a good  
idea...

It would be even better if we could ask the switch what our port  
supports, but I'm not sure how to do this in such a way that a switch  
that doesn't support this protocol floods the request so the results  
are meaningless.
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Ryota Hirose <[email-masked]>
Date: Mon, 25 Jul 2005 19:02:09 +0900 (JST)

Hi,

>From: Bob Hinden <[email-masked]>
>Date: Fri, 22 Jul 2005 13:18:24 -0700

> just a bad idea.

I think so, but in a real world, such bad ideas are often accepted.

In the IPv4 environment, MTU problem is more simple because DHCP has
no MTU option.  The mechanisms to adjust packet size are PMTUD and TCP
MSS negotiation.  For TCP applications to the Internet, these
mechanisms will work fine.  For UDP or ICMP applications to the
Internet, almost applications restrict a packet size less than 1500.
Of course, NFS uses large packet size, but almost people will not use
NFS over the Internet.  So, people can use the Internet fine, and
enjoy Jumbo Frames in LAN.

To make a same situation for IPv6, we must not include MTU option in
RA.  An interoperability will be down slightly, but we can use Jumbo
Frames.  If RA includes MTU option, we will get a perfect
interoperability about MTU, but cannot use Jumbo Frame.

Now, people are accumulating the knowledge about Jumbo Frames.  They
will learn the trick to use Jumbo Frames for IPv4, and will say, "IPv6
is slower than IPv4, because it disables Jumbo Frames...".

I think there are three ways for us.

1) Do nothing.  Jumbo Frames is an illegal specification for IEEE
   802.3, and there is no de facto frame size also.  Wait until IEEE
   will move, or a de facto standard will be made.

2) Make a new protocol for MTU/MRU discovery/negotiation.  Thanks for
   Mark, Iljitsch or others.

3) Make a memorandum for Jumbo Frames with current implementations.
   Fix the implicit maximum value of RFC2464, how to enable Jumbo
   Frames for IPv6, the estimated problems, the implimentation
   restrictions, and so on.

Ryota Hirose
Yamaha Corporation

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 25 Jul 2005 14:59:11 +0200

On 25-jul-2005, at 12:02, Ryota Hirose wrote:

>> MTU's is

:-)

Thinking a bit more about this: ethernet rules the world right now,  
and I see few competitors on the horizon. Although an ethernet switch  
detects attachment to a port and even negotiates a few capabilities,  
there is no mechanism to furter distribute the information learned  
that way (if any), so the only way to be certain that all systems  
connected to an ethernet use the same MTU is by observing the 30 year  
old 1500 byte maximum, or by having an administrator police the MTU  
of all attached devices. In other words: the current situation, where  
jumboframe deployment is negligible.

I think we have to turn this around and acknowledge that having a  
fixed MTU on any network (whatever the layer) stifles progress so we  
should embrace MTU diversity.

---

From: Bob Hinden <[email-masked]>
Date: Sat, 23 Jul 2005 17:17:25 -0700

Iljitsch,

>equal jumboframe capability, so the problem remains.

Right, I don't know what the correct designation is either.

>with is usable in practice.

Fair point.  As suggested on the list a new NA/NS option seems like a 
reasonable approach except for the problems pointed out.  Having to do 
probes of different packet sizes is fairly messy if there is a device in 
the middle with a smaller MTU.  Would a NA/NS option solution solve enough 
of the problem to make doing it worthwhile?

Bob

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 26 Jul 2005 00:00:20 +0200

> Fair point.  As suggested on the list a new NA/NS option seems like  
> a reasonable approach except for the problems pointed out.  Having  
> to do probes of different packet sizes is fairly messy if there is  
> a device in the middle with a smaller MTU.  Would a NA/NS option  
> solution solve enough of the problem to make doing it worthwhile?

If in a conservative approach we don't want to send "jumboprobes" we  
still need an easy way to let the hosts know the limits of the  
infrastructure. A packet that is multicast by systems "in the know"  
would do the trick. Since the most obvious source of knowledge would  
be switches, it makes sense to make it easy for a switch to multicast  
this information, which means it shouldn't be part of RAs as switches  
as a rule don't send those out. (If we make the message very simple  
and let the switch use the unspecified address as a source address,  
there wouldn't be any need for the switch to run IPv6, it would just  
transmit a fixed content packet.)

In the absense of "smart" switches, one or more routers or even hosts  
could be configured to send out proxy jumboframe size announcements.

When a host hears multiple jumboframe sizes announced, it would  
select the smallest it hears.

To avoid the situation where there is a smart switch that announces a  
jumboframe size and also a stupid switche that doesn't, and also  
don't support jumboframes (of that size), or the situation where an  
administrator makes a mistake, it would still be necessary to confirm  
that the advertised MTU size is correct by sending an MTU sized  
jumboframe after an ND option indicates that a neighbor supports  
jumboframes.

This approach still provides most of what we'd like, except that all  
systems share the same effective MTU, which they learn from an  
administratively configured value, and hopefully in the future,  
directly from the layer 2 infrastructure.

What do you think?

---

From: Mark Smith <[email-masked]>
Date: Tue, 26 Jul 2005 12:11:43 +0930

Hi Bob, Iljitsch,

On Sat, 23 Jul 2005 17:17:25 -0700

> of the problem to make doing it worthwhile?

I'd like to suggest a two phased development approach, based on whether
layer 2 can be assumed to fully support the larger MTUs or not (I think
the following is also probably a summary of the couple of threads of
discussion in the emails of the last couple of days) :

(1) Making the assumption that layer 2 can support larger, non-standard
MTUs (as it has been engineered to by a network designer), develop
mechanisms that allow nodes to announce their non-standard, larger MTU
support. If a pair of nodes find they both support larger MTUs/MRUs,
then they'll take advantage of it for unicast traffic. If the unicast
communcations fails because layer 2 doesn't support the MTU/MRUs it is
supposed to, then layer 2 is broken and it is up to network admin to
fix it.

In all other cases (eg. multicast, unicast without these announcements),
the MTU used will be the either the link layer MTU standard or the link
MTU value as announced in RAs (i.e., along the lines that this MTU value
has been announced because the link layer technology doesn't have a
standard value e.g. token ring. There is a corner case that needs to be
covered where a larger than standard MTU has been annnounced for a
technology that does have a fixed MTU e.g. 1500 byte ethernet, and a
sub-set of nodes only support the standard size).

These mechasims only come into play if non-standard MTU support has been
enabled (in some mechanism specific manner, which may be via a new RA
option, or even just configuring larger, non-standard  MTUs on the
interfaces that are capable of larger frames). The out-of-the-box
plug-and-play IPv6 functionality that exists today will be preserved if
these mechanisms aren't enabled, by all nodes using the link standard MTU
or RA announced MTU. 

I think this solution would be relatively quick and simple to develop,
and is applicable to any networks that have been specifically designed
to support larger, non-standard MTUs. 

(2) Develop mechansisms that can dynamically discover MTU/MRU sizes,
limitations and variations over time, including within intermediary
layer 2 devices e.g., switches. It seems that some solutions have
already started to be developed in this area, including the email
discussion between Iljitsch and Perry, Matt Mathis in
draft-ietf-pmtud-method-04.txt and Fred Templin in
draft-templin-ipvlx-02.txt.

If the above phased approach is followed, I think it would be useful to
allow any ND  or other options developed for phase 1 to be re-used for a
phase 2 solution if they can.

Any thoughts on this approach ?

Regards,
Mark.

---

From: Perry Lorier <[email-masked]>
Date: Tue, 26 Jul 2005 20:41:39 +1200

> supposed to, then layer 2 is broken and it is up to network admin to

It's difficult to say what MTU L2 should support.  If you have a GigE
link then you might assume 9k MTU's (which varient of 9k?), and the
remote end also might have GigE, but there is a 100mbit switch in the
middle preventing communication between both ends to use 9k MTU's.

You can barely assume 1500 as people tend to tunnel things, you can get
technologies like vlans, 802.1x or mpls eating heavily into your mtu.
You should assume 1280 bytes and hope that you can raise it.

More and more end users aren't qualified network admins, they are people
who plug device A into device B and expect everything to "just work".
Given how many PMTU failures you see on the production internet today,
assuming even qualified network admins have the time and resources to
debug current MTU mismatches seems unlikely let alone future ones.

Any system which requires admin intervention to work IMHO will be a
dismal failure.

> sub-set of nodes only support the standard size).

For multicast it seems wise to just "give up" and use 1280.  Even if you
can determine the MTU of all nodes in a multicast group, there are no
assurances that it is going to remain stable and changes are much harder
to deal with than the unicast case.  Maybe allow an admin to tune this
higher if they want to.

> Any thoughts on this approach ?

If some options can be found for ND which significantly improve the
situation in (1), then by all means :)  I personally think that we
should resign ourselves to the fact that a single L2 segment having the
same MTU is less likely to occur in future, rather than more likely so
any attempts to try and find a unified MTU for a link is unlikely to be
profitable.

---

From: Stig Venaas <[email-masked]>
Date: Tue, 26 Jul 2005 11:17:30 +0200

> You should assume 1280 bytes and hope that you can raise it.

You can't really assume 1500 on L3, but PMTU discovery should work
there. On L2 I thought you generally had 1500 despite vlans and 1x.

> to deal with than the unicast case.  Maybe allow an admin to tune this

PMTU discovery does work for IPv6 multicast (you're supposed to get
packet too big ICMP messages) and at least implementations I'm familiar
with do indeed support it. It is less stable since the PMTU might change
when people join or leave the group. With respect to L2 it's not an
issue as long as the hosts interface is configured to a common L2 MTU.
Supporting different MTUs for different hosts on the same link sounds
rather scary IMO.

> any attempts to try and find a unified MTU for a link is unlikely to be

My personal perhaps naïve thinking is that the administrator should
configure all hosts on the link to a common MTU, and ND RAs would
have been a fine tool for that. But according to 2464 it seems it
must be configured in some other way.

Stig

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 26 Jul 2005 13:05:31 +0200

> supposed to, then layer 2 is broken and it is up to network admin to

You mean: if I hook up two hosts that support 9k jumboframes to a  
cheap unmanaged 10/100 switch (all out of the box) and it doesn't  
work, I have to go in and reconfigure the hosts so it does work?

Sorry, but that's not acceptable.

And the case where I have to enable jumboframes on the hosts and then  
everything fails because the switch can't handle it is barely usable.  
It allows for jumbo and non-jumbo capable hosts to live on the same  
subnet, which is an important advantage, but it's still extremely  
fragile for no good reason.

What we need in addition to this would be (as outlined in my message  
to Bob):

- a mechanism to distribute the MTU for the layer 2 network to jumbo- 
capable systems
- validation that jumboframes indeed work to minimize the impact of  
misconfiguration

> MTU value as announced in RAs

Right.

> There is a corner case that needs to be

If the nodes announce their MTU/MRU in neighbor advertisements this  
isn't a problem: jumbo-capable hosts wouldn't send jumboframes to  
neighbors that don't support them.

> enabled

Yes, this is important.

> (in some mechanism specific manner, which may be via a new RA
> option,

Suboptimal because switches can't easily do this.

> or even just configuring larger, non-standard  MTUs on the
> interfaces that are capable of larger frames).

Very suboptimal because it requires manual configuration of all nodes.

> The out-of-the-box
> these mechanisms aren't enabled,

Yes, and the destructive potential of nuclear bombs is mitigated if  
you don't detonate them... This is 2005 and IPv6. Autoconf is the  
word, is the word that you heard. It's got groove, it's got meaning.  
Autoconf is the time, is the place is the motion. Autoconf is the way  
we are feeling.

> discussion between Iljitsch and Perry,

:-)

We can certainly experiment with this while we roll out the more  
conservative approach.

> Matt Mathis in

I don't really see how those apply.

Of course.

---

From: Perry Lorier <[email-masked]>
Date: Tue, 26 Jul 2005 23:46:47 +1200

> Agree, except that packets are cheap on a 1000 Mbps LAN, so those  don't
> count much towards 6.

Packet rate however starts becoming a problem at faster speeds, at gige
it starts becoming a problem for hosts to deal with unless they are
careful.  And not all networks are fast, 3G networks are becoming more
prevalent.  We should not waste resources needlessly :)

> That's not what the mechanism that I outlined does. What it does do  is
> let all jumbo-capable system send send at least one packet at  their
> maximum size and get feedback on whether it was received by  anyone. For
> that, every packet may update information that the system  (host or
> router) holds, but then the old information is gone so there  is no
> per-(potential) correspondent state.

Ok, I sat down and reread it a few more times and I think I've got more
what you're trying to get at now.

> Hm, they invented routers for a reason.

You know, that was exactly my thought when I heard about it too. :)

> I don't think we have to bend over backwards to accommodate unreasonably
> large layer 2 networks. But: what would be reasonable to accommodate?
> 1000 systems?

I think it's probable that 1000 systems would be quite feasible.  It's
also possible that as the scarcity of addresses is relieved that people
move back towards IP based virtual hosting.  How many SSL sites would a
single ISP want to host if it could do so easily?

If you have a big outdoor concert with some AP's covering it you could
have thousands of people all ending up sharing a L2.

> far) switches use per-port buffer space, so although such a packet  uses
> up buffer space for a lot of ports, there is nothing  unreasonable about
> that (packet is gone again in 72 microseconds  anyway). But some vendor
> feedback would be good here.

Fair enough.

> Neighbor discovery fails?

Host A can talk to Host B ok.
Host A can talk to Host C ok.
Host B can't talk to Host C.

This happens in ad hoc wireless networks.  With your system I'm not
entirely sure how you deal with who's "turn" it is next if not all nodes
can see all other nodes.  Host A should still be able to talk to Host B.

> I don't think those exist in that way.

Ad hoc wireless was a much better example ;)

> Ah, but how do you know that A doesn't talk to D, and is never going to?

How do you know it will in the time before the topology of the network
changes?  Given that the topology of the network changes every time a
host comes and goes, the chance that you'll want to talk to most of the
users during time is rather low.

> When ND indicates a neighbor supports jumboframes we start by  sending
> an MTU prope with either an MTU that worked towards another 
> receives this packet it sends an ack and both sides can increase the  MTU.

I'd start at the minimum "MTU" size.

A colleague of mine (Matthew Luckie) has done some research into path
MTU's.  He has a work in progress paper (
http://www.wand.net.nz/~mjl12/debugging-pmtud.pdf ) where he enumerates
all the common MTU's he's seen on the Internet.

I'd start with a similar table trying the lowest size, and sending that,
if it's received try the next lowest size and so on until you don't get
a reply.  When you don't get a reply try the previous-mtu-that-worked+1,
if that succeeds start a binary search between previous-mtu-that-worked
and the one that didn't.

For a "common" MTU, you only have to endure two timeouts (the next
highest common MTU, and the +1 test). For an uncommon MTU you can
increase the MTU to maximum "common" MTU that's lower than your MTU
quickly, and can endure the timeouts from then on.

If you pick a common MTU and hope, you end up having to endure timeouts
in the case of an incorrect guess.

If you don't want to hard code that table into your stack, you can at
least do something simple like increment the probed MTU by say 512.

> minimum of local and remote MTU/MRUs), at one packet per second or so.

You want to try and avoid doing a binary search because about 50% of the
time, you're going to have a timeout.  log2(9000-1500)>10, thats 10
timeouts, if each timeout is a second or so as you suggest, then that's
over 10 seconds for a successful mtu probe.

If you start low and move higher then you'll get a successful response
from the remote end immediately, increasing your MTU every round trip.
As soon as you get a timeout, you hope that the MTU you tried was the
previously successful "common MTU", so you try that one +1, to make sure
 it fails.  If it succeeds then it's a completely "unknown" MTU so only
then do you fall back to a binary search.

[later after reading below], hrm you're suggesting sending two packets
at once, which is fine this so long as reordering doesn't occur (which
seems unlikely at L2).

> nearly all MTUs are 32 bit aligned, the maximum 3 extra bytes aren't 

Yep. See the table I quoted above.

> I think we can assume the MTU is the same in both directions. 

I agree.

> So when system A tries with 3000 bytes (worked with C!) towards B, B sets an 
> with 6108, and so on. 

Hang on, if they don't receive a packet, how can they know to send a
NAK?  if they're just waiting for a timeout how can they know if the
packet got lost on the way there or on the way back?

> With each successful packet (either received or

Yep!  So you want to increase your chances of successful packets getting
through :)

> You can do this with binary search too, as long as you send a  separate
> "now testing YYYY, ack XXXX" packet.

Doubling the number of packets that need to be sent.  Chances are it's
going to be one of a very few sizes, and as you say, it'll probably even
be the size that the other ones on the link are.

If instead of using special "ICMP MTU Probes" we use "ICMP Echo request"
/"ICMP Echo Reply" messages, there is no changes to any packet formats
needed, all it needs to be done is have implemented in a TCP/IP stack,
and the concept is even reusable for IPv4.  Other hosts don't even have
to be upgraded to support this either.  magic!

Stacks would be free to do as you suggest (doing a binary search) or as
I suggest (ramp up and do a binary search only as a last resort).

So the general approach would be:
* If a packet arrives from a host that is larger than the cached MTU for
that neighbour, increase it to the size of the packet arriving.
* When receiving a ND (but not a NS!), and you have no cached MTU for
that neighbour, you start the MTU discovery process (using any mechanism
for selecting the packet sizes the implementation deems appropriate (ie,
either yours, or mine, or if someone can come up with a method thats
even better than ours, they could use that!)

> control traffic could get in the way of real traffic.

Hrm, now thats a good point. Ah, but you should only be putting one
packet onto the network after another packet has come off, so you can't
overflow any queues with it.

> If the "I sent..." packet comes in before the actual test packet,  then
> this would look like the test packet didn't make it, so the  receiver
> would send a NAK. However, if the packet does make it and  comes in
> late, then obviously the receiver notices this and it can  send out an
> ACK as well. Since we don't want to hammer the layer 2  network with
> possibly invalid packets, the receiver (well, sender of  the original
> probe) would probably want to wait long enough for that  ACK to come in
> before sending the next packet anyway.

Hammering shouldn't be too bad -- You're only putting a packet onto the
network when a packet is taken off it, so you can't overflow any
buffers.  With packet rate issues, any host which is "slow" just slows
down the rate of packets being sent to the rate that it can cope.

> Yikes! You really are a modern day René Descartes, aren't you?  :-)

Well, this needs to work on L2's that aren't Ethernet (even if there
aren't many of them left!) so assumptions that everything has a MAC may
be premature.

> Ring networks are very cool, too bad we don't have them anymore.

Heh, indeed. :)

> Well, that's what administrators do, isn't it?

Not when it's my flatmate who wants to plug his shiny new console into
the switch and have it "just work".  Even if I understand MTU's I really
don't want to start trying to figure out what MTU's all the switches on
a large campus are.  What was the MTU of the 3 port switch in my VoIP
phone?  Who knows?  And to be perfectly honest, who the hell cares? :)
I should plug everything in and it should "just go" and have the best
possible performance.

> anything to do with "I can handle 1500".

Which switch?  I live in a flat with 3 other people, we have at least 4
devices that act like switches on one segment.  (2 switches, a voip
phone (you can daisy chain a PC off it), and an AP).  I have no idea
what the maximum MTU of all those switches are, let alone all the end
hosts around here.  Sure I could spend an afternoon configuring
everything just right so that I get the maximum possible efficiency
through my network, but it's unreasonable to think that everyone else in
the world will.

> protocol, but not as RA options. That way, switches can announce  their
> own MTU capabilities, even if they don't otherwise support  IPv6. So if
> the switch says that it can do 4500, we only have to try  4500 (ack) and
> 4504 (nak) and everything is much faster. (Unless the  layer 2 network
> is more complex, of course, but then either 4500 gets  a nack/timeout or
> 4504 gets an ack.)
> 
> Hm, maybe 4 bytes larger than an earlier maximum is always a good  idea...
> 

Hrm, so Ethernet has capability negotiation (which is how speed, duplex,
pause frame support etc is negotiated).  I have no idea if it says if
the switch supports jumbo gram, IEEE specs make my head hurt.

[hrm, this email is getting rather long :)

---

From: Mark Smith <[email-masked]>
Date: Tue, 26 Jul 2005 22:36:20 +0930

Hi Iljitsch,

On Tue, 26 Jul 2005 13:05:31 +0200

> Sorry, but that's not acceptable.

I think that is an uncommon scenario. How many hosts with jumbo frame
capable interfaces come with jumbo frame capablility _enabled by
default_ ? The complex "cope with any MTU on any device" solution is
really trying to address what I think would currently be the worst case
scenario. I don't think that scenario will be all that common, at least
in the short to medium term because :

(a) devices that are jumbo frame capable don't come with them enabled
by default

(b) people who enable jumbo frames usually know what they're doing and
what is required for them to work ie. a layer 2 infrastructure that will
support them.

If you enable jumbo frames on end-nodes without a layer 2 infrastructure
that supports them, then you get what you deserve - a network that won't
work. The solution of course is go back to standard size frames or buy
jumbo capable layer 2 infrastructure and enable it. I think it's really
as simple as that.

Don't fall into the trap of assuming the worst case scenario will be the
common one. I think the common one is that people who specifically buy
_and want to enable_ jumbo frame capable end-nodes will also make sure
they buy jumbo frame capable layer 2 infrastructure.

> fragile for no good reason.

See above. You broke the network by enabling a feature on your end-nodes
that your network doesn't support.

> - a mechanism to distribute the MTU for the layer 2 network to jumbo- 

There are probably a few RA announced-style MTU values that need to be
distributed around :

(a) the original MTU that is currently in RAs - the "whole of link" one.
This is the MTU that all nodes on the link must be capable of receiving,
as it will be used as the maximum size for multicasts. Not really
necessary for ethernets as this value would normally be 1500 bytes,
although there are some uses for ethernets if all offlink destinations
are via a link with a slightly smaller MTU e.g., and IPsec tunnel, to
avoid a PMTUD cycle. The other reason to specifically set it is to cope
with layer 2 technologies such as token ring which don't have a
standardised MTU.

(b) the "link maximum jumbo" ie. the absolute largest MTU that the layer
2 infrastructure can support. Nodes that can support jumbo frames can
raise their MTU up to either the maximum MTU they support above the
standard value that is less than this value (eg 7k if they don't support
9K jumbos (they exist, even in fast ethernet. My Netgear Fa312s under
Linux support MTUs of 2024 bytes !)) or set their MTU to this value, which
may actually be smaller than the jumbo size they're capable of
supporting (I think I've read that some Intel GigE NICs can support 16K
jumbos). This value would be configured on the routers making RA
announcements after the layer 2 infrastructure has been had jumbo frame
capability enabled.

> - validation that jumboframes indeed work to minimize the impact of  

Once your layer 2 infrastructure is known to support a specific jumbo
frame size, I don't think it is all that necessary to check for it
anymore. The overheads of checking for it all the time may be too high
when compared to how often unauthorised standard MTU or unconfigured
switches are plugged in to the network.

> > sub-set of nodes only support the standard size).

I think my "link jumbo maximum" description previously provides a
solution to this corner case.

> neighbors that don't support them.

Agree.

> Suboptimal because switches can't easily do this.

Which is why I'd have the routers do it in their RAs.

> Autoconf is the time, is the place is the motion. Autoconf is the way  

The problem is going too far with the goal of PnP, and then ending up with
monsterously complex solutions that attempt to address every possible
scenario, including the most esoteric onces, rather than just the common ones
(ie. common operational scenarios, common failure scenarios).

> > discussion between Iljitsch and Perry,

Don't get me wrong, I'd like this solution to be developed. I'm just
wondering how hard it might be to develop, and if there is simpler
interrim "sub-solution" that would still be useful to a fair number of
people, namely those who don't expect to just "plug-and-play" when they
want to use non-standard protocol parameters and feature, such as jumbo
frames.

Once a fully fledged "cope with any MTU" solution exists, then
manufacturers of NICs and switches could enable jumbo frames by default.

> I don't really see how those apply.

Only because from what I remember of Matt's draft, and what Fred said
about his, they perform some sort of MTU discovery without relying on a
feedback mechanism e.g. ICMP Dest Unreachable, Packet Too Big, which I
think you solution would have to deal with, assuming that there wasn't a
requirement to add protocols to the switches.

Regards,
Mark.

---

From: Perry Lorier <[email-masked]>
Date: Wed, 27 Jul 2005 01:22:58 +1200

> they buy jumbo frame capable layer 2 infrastructure.

If you're going to require admins enabling jumbograms on their network,
it's never going to happen.  People are just going to avoid enabling it
because when you do everything breaks.  As it stands today the general
consensus is "disable jumbograms, coz even if you manage to get it all
working today, something will change tomorrow and you'll have a nasty
problem to fix".

If a simple solution can be found to having one link support multiple
MTU's then thats where I think we should be heading :)  If one can't be
found, then we should investigate what we can do to make this as easy as
possible for end users.

---

From: Francis Dupont <[email-masked]>
Date: Tue, 26 Jul 2005 17:34:48 +0200

In your previous mail you wrote:

   
=> this is not acceptable because jumbo frames are too good for
performance (two years ago I discover my FreeBSD (again :-) TCP
ran at a speed proportional to the frame size up to ~1Gbits/s).

=> I disagree: this should be the job of IEEE, not the IETF or the
IPv6 WG one.

=> I vote for this. In fact, it seems we have only/first to fix
the wording, i.e., to replace the default MTU from the standards
by the default MTU configured on the interface, haven't we?

Regards

[email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 26 Jul 2005 17:48:54 +0200

Hi Mark,

It's clear that we're thinking from two very different starting  
points. You want to make a few very small changes to the current way  
of using jumboframes, all the while accepting all the current  
limitations that make that virtually nobody uses them.

What I want to do is remove the limitations so jumboframe capability  
will be used whenever it's available.

> as it will be used as the maximum size for multicasts.

Right. We leave this one alone.

> 2 infrastructure can support.

Yes.

> (I think I've read that some Intel GigE NICs can support 16K
> jumbos).

I was once testing switches from a vendor we hadn't used before, and  
I complained that they didn't support jumboframes. A few days later  
the rep came by again with a card that supported 64000 byte frames.  :-)

> This value would be configured on the routers making RA

Yes, except that it would be even better to have _switches_ announce  
this value since they're the ones that know it, which means a new  
message type.

> anymore.

People can hook up extra layer 1 or layer 2 devices to your jumbo- 
capable switch. It's nice if that doesn't bring down the network.

> The overheads of checking for it all the time may be too high

One extra packet in each direction whenever an ND exchange takes  
place doesn't seem excessive, especially on gigabit ethernet.

Remember that we routinely encrypt untold terabytes of data on the  
off-chance that someone may look at it some time.

>> Autoconf is the word, is the word that you heard. It's got groove,  
>> it's got meaning. Autoconf is the time, is the place is the  
>> motion. Autoconf is the way we are feeling.

> (ie. common operational scenarios, common failure scenarios).

"There is no way anyone will be using this software by the year 2000  
anyway..."

All of this really isn't very complex. All that's needed is:

Simple switches:

- periodically multicast a packet with fixed content

Routers and managed switches:

- accept a user configured value for each interface and multicast an  
IPv6 packet containing that value

Jumboframe capable hosts and routers:

- listen for a new type of message to learn link jumbo MTU
- whenever such a message is received, see if it's lower or equal to  
the current stored value, update timestamp
- remove stored jumbo MTU for an interface when it's not refreshed in  
the last N seconds

- update IPv6 MTU for interface based on learned jumbo MTU / local  
maximum
- put current IPv6 MTU in neighbor advertisements/solicitations
- when receiving neighbor advertisement/solicitation containing jumbo  
MTU option, send test packet at mimimum of local/remote jumbo MTU
- when receiving test packet, update IPv6 MTU for neighbor
- when NUD is triggered, update IPv6 MTU for neighbor to regular IPv6  
link MTU

This is a very simple implementation that doesn't handle lost packets  
and optimizing for the largest possible jumboframe size, but it  
should work well enough.

> manufacturers of NICs and switches could enable jumbo frames by  

No, this is exactly the wrong thing to do. The overhead of making ANY  
change is so huge that it's really not worth it unless you get it  
right. Also, there often isn't a second chance. Getting things right  
the first time in actual deployment is very important.

Iljitsch

---

From: "Stephen Sprunk" <[email-masked]>
Date: Tue, 26 Jul 2005 10:48:39 -0500

> I think that is an uncommon scenario. How many hosts with jumbo
> frame capable interfaces come with jumbo frame capablility _enabled
> by default_ ?

They can't today according to the existing RFC.  What we're proposing is a 
solution that works in any environment with zero configuration so that 
jumbos _can_ be enabled by default.

> The complex "cope with any MTU on any device"
> solution is really trying to address what I think would currently be the
> worst case scenario. I don't think that scenario will be all that
> common, at least in the short to medium term because :
>

Neither of these statements (and the snipped discussion that follows) fit 
the model some of us are targeting, i.e. Joe Sixpack plugging a bunch of 
stuff together at home.  IMHO, he should get the benefit of jumbos when 
possible even if he has no clue what "IP" means, much less "jumbo frames".

And, I hate to point it out, but even many "professional" admins out there 
fall into the Joe Sixpack category.

>> What we need in addition to this would be (as outlined in my
>> message to Bob):
>>
>> - a mechanism to distribute the MTU for the layer 2 network to
>> jumbo-capable systems
>
> There are probably a few RA announced-style MTU values that need
> to be distributed around :
>

I think I'd name this "link multicast MTU" or "link minimum MTU" and I'd 
default it to 1280 on all media types unless a larger value was advertised 
(remember, there may not be a router on all networks, and many routers won't 
implement jumbos at all).

> (b) the "link maximum jumbo" ie. the absolute largest MTU that the
> layer 2 infrastructure can support. Nodes that can support jumbo
> frames can raise their MTU up to either the maximum MTU they
> support above the standard value that is less than this value (eg 7k if
> they don't support 9K jumbos (they exist, even in fast ethernet. My
> Netgear Fa312s under Linux support MTUs of 2024 bytes !)) or set
> their MTU to this value, which may actually be smaller than the
> jumbo size they're capable of supporting (I think I've read that some
> Intel GigE NICs can support 16K jumbos). This value would be
> configured on the routers making RA announcements after the layer
> 2 infrastructure has been had jumbo frame capability enabled.

I'd call this "link maximum MTU", which would be used to constrain the upper 
bound of the MTU search space.  The upper bound might also be constrained by 
a host with a lower hardware MTU.  In the case where there's no router 
sending such an advertisement, the upper bound would equal the host's 
hardware MTU.

Note that in all cases, a host's MRU should always be the maximum that the 
hardware supports.  It's only the mcast and per-neighbor MTUs that are 
lower.  "Be liberal in what you accept" and all that.

> switches are plugged in to the network.

Methinks you overestimate (a) the number of cases where the "admin" knows 
the MTU, and (b) the odds that what they "know" is correct, either 
immediately or after a couple years of maintenance.

I don't think a probing scenario is avoidable, unfortunately.  If people 
make one small mistake and the hosts don't detect and work around it, the 
network fails; people will learn that jumbos "don't work" and rarely use 
them.  And, if you start with the assumption you need to probe, the solution 
becomes more obvious and the problems you can solve become more interesting 
and compelling.

> people, namely those who don't expect to just "plug-and-play" when
> they want to use non-standard protocol parameters and feature,
> such as jumbo frames.
>

And at least some of us appear to be trying to solve the larger problem now 
instead of addressing it in two stages as you propose.  I don't like the 
idea of half-solutions because it delays getting to the real goal, and the 
interim step seems just as bad if not worse than the status quo.

>> > Matt Mathis in draft-ietf-pmtud-method-04.txt and Fred Templin
>> > in draft-templin-ipvlx-02.txt.
>>
> Only because from what I remember of Matt's draft, and what Fred
> said about his, they perform some sort of MTU discovery without
> relying on a feedback mechanism e.g. ICMP Dest Unreachable,
> Packet Too Big, which I think you solution would have to deal with,
> assuming that there wasn't a requirement to add protocols to the
> switches.

Any solution which requires an upgrade of all the L1/L2 devices in a path 
will fail in the majority (IMHO) of cases.  Let's stick to smart edges and 
dumb networks.

S

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 26 Jul 2005 18:46:37 +0200

> prevalent.  We should not waste resources needlessly :)

Well, the places where jumboframes are worth the trouble are also the  
places where a handful of packets won't make a difference. I'm not  
sure how fast 3G is, but I believe not more than a few Mbps, so  
jumboframes really aren't very useful there because they occupy the  
channel for too long. Doubly so on radio networks with their high bit  
error rates.

> can see all other nodes.  Host A should still be able to talk to  

Well, a simple way to decide could be a log of the difference in MAC  
address. So after host 20 sends its packet, host 28 would wait for 3  
seconds and host 36 for 4 seconds. But host 36 hears host 28 and  
resets its timer to 3 seconds. If hosts 28 and 36 can't hear each  
other, host 36 will send its packet 1 second after host 28 rather  
than 3 seconds. No big deal.

> users during time is rather low.

Look at it this way: if two routers send out RAs every 10 seconds,  
that's one packet every 5 seconds. If 60 hosts all send one packet  
every five minutes, that's also one packet every 5 seconds.

> I'd start at the minimum "MTU" size.

Yes, I thought about this and first trying a 1508 byte packet makes  
sense: if jumboframes don't work, you've wasted as little time and  
bandwidth as possible. If they do work, you've only wasted 1508 bytes.

> all the common MTU's he's seen on the Internet.

And reaches a very interesting conclusion! Exchanging per-neighbor  
MTUs would really help here.

> if that succeeds start a binary search between previous-mtu-that- 

I partially agree. If you're at a well known boundary and want to  
search upward, it makes sense to try that well known boundary +  
minimum increment (I say: 4) first. That way, if you can't go beyond  
the current boundary, you know so immediately. Next is the highest  
possible value. If you can use that one, you're done.

But if previous low + minimum works but maximim doesn't, a mostly  
binary search still makes sense. However, it could be a "hinted"  
binary search. For instance, if you're searching between 1508 and  
9000 (with the target being 4464) a strict binary search would do:

1  1508 yes
2  9000 no
3  5252 no
4  3380 yes
5  4316 yes
6  4784 no
7  4548 no
8  4432 yes
9  4488 no
10 4460 yes
11 4472 no
12 4464 yes
13 4468 no

A hinted binary search could be:

1  1508 yes
2  9000 no
3  4470 no (closest value to binary 5252 target)
4  2048 yes (closest value to binary 2988 target)
5  2052 yes (see if 2048 was our limit)
6  4352 yes (closest value to binary 3260 target)
7  4356 yes (see if 4352 was our limit)
8  4464 yes (closest value to binary 4412 target)
9  4468 no (4464 was our limit)

Note that although the second variant is faster overal, the first one  
finds a reasonable candidate (that can already be used at that point)  
at try 5, and the second one at try 6.

In this case your serial bottom-to-top search would probably be a bit  
faster, but it has two disadvantages: it takes a long time to find a  
high MTU, and it's not good at finding non-standard MTUs.

> quickly, and can endure the timeouts from then on.

Note that with a 100 ms timeout (more than enough) you're done in  
less than 2 seconds worst case.

> packet got lost on the way there or on the way back?

Good question.  :-)

> to be upgraded to support this either.  magic!

You mean, rely just on ICMP and not announce a bigger MTU in RAs?

I guess you're right, but I wouldn't want to be a 10 Mbps host in an  
otherwise 64k jumbo-enabled network, because all those probes would  
eat up my bandwidth even though I can't successfully receive them.

Also, I think we want to be nicer to on-link probers than off-link  
ones, especially with these large packets.

> I suggest (ramp up and do a binary search only as a last resort).

Yes, this can be left up to the implementers.

> that neighbour, increase it to the size of the packet arriving.

Not sure if we want to do this check for every packet. Also, an  
attacker could fake the packet in order to do an "MTU attack" on a  
non-jumbo enabled host.

> even better than ours, they could use that!)

With an MTU option in it. And why not NS?

> what the maximum MTU of all those switches are

If all of those switches announce their MTU, we're in business.

On the other hand, if we do an MTU search we don't need this  
information because we'll find out ourselves.

If we don't do an MTU search and the switches don't announce their  
MTU, you're probably not going to use jumboframes on such a network...

> the switch supports jumbo gram, IEEE specs make my head hurt.

Autonegotiation only does 16 bits or something like that, no room to  
include the MTU there. Gigabit does have some in-band stuff like flow  
control, maybe that can be reused. But you always run the risk that a  
dumb switch just forwards those packets and screws up the negotiation.

---

From: Perry Lorier <[email-masked]>
Date: Wed, 27 Jul 2005 22:54:51 +1200

[Bugger! Lost the reply I was writing for this! ]

> places where a handful of packets won't make a difference. I'm not  sure
> how fast 3G is, but I believe not more than a few Mbps, so  jumboframes
> really aren't very useful there because they occupy the  channel for too
> long. Doubly so on radio networks with their high bit  error rates.

Good point :)

> seconds and host 36 for 4 seconds. But host 36 hears host 28 and  resets
> its timer to 3 seconds. If hosts 28 and 36 can't hear each  other, host
> 36 will send its packet 1 second after host 28 rather  than 3 seconds.
> No big deal.

Yeah, but if this is happening lots you will end up with lots of hosts
transmitting close together.  Maybe it's not a problem as you suggest,
but I can't help feel that this seems a bit too complicated.

> every five minutes, that's also one packet every 5 seconds.

If they send it in the same 5s tho you'll melt your queues :)

How long will this take to converge?  Remember that you have to start
again as soon as the topology changes (eg a new host is added/removed).

The advantage with this part of your scheme is that it does give you
MTU's that you can use for multicast.

> bandwidth as possible. If they do work, you've only wasted 1508 bytes.

I'd start with 1500; check your assumptions.  In fact, it might be worth
starting at 1280 and if that fails log a message that the L2 is broken.

There are enough networks out there that use various tunnels that the L2
MTU is slightly smaller than you would expect to make it worth while to
at least check.

> And reaches a very interesting conclusion! Exchanging per-neighbor  MTUs
> would really help here.

Exactly.  :)

>> a reply.  When you don't get a reply try the previous-mtu-that- worked+1,
>> if that succeeds start a binary search between previous-mtu-that- worked
>> and the one that didn't.
> 
> search upward, it makes sense to try that well known boundary +  minimum
> increment (I say: 4) first. That way, if you can't go beyond  the
> current boundary, you know so immediately. Next is the highest  possible
> value. If you can use that one, you're done.
> 
> binary search still makes sense. However, it could be a "hinted"  binary
> search. For instance, if you're searching between 1508 and  9000 (with
> the target being 4464) a strict binary search would do:
> 
> 1  1508 yes
> 2  9000 no
> 3  5252 no
> 4  3380 yes
> 5  4316 yes
> 6  4784 no
> 7  4548 no
> 8  4432 yes
> 9  4488 no
> 10 4460 yes
> 11 4472 no
> 12 4464 yes
> 13 4468 no
> 

Mine would do:

1  1514 yes
2  1536 yes
3  2002 yes
4  2048 yes
5  4352 yes
6  4464 yes
7  4470 no
8  4465 no

Now, if you assume a timeout is at least 2 RTT's (and I suspect it
should probably be more) mine takes 6RTT's to find the right MTU, and
takes 10RTT's to prove it.

Your first one takes 19 RTT's to find it and 20 RTT's to prove it.
Your second (hinted) one takes 10 RTT's to find it, and 12 RTT's to
prove it.

As the MTU's get larger yours gets faster (since it is less likely to
hit a "no") and mine gets slower.

Mine is slow if there is a non standard MTU (which I suspect would be
rare, but anyway).  But after it's found a non standard MTU it can be
added to the table and it'll be found quickly next time.

> Note that with a 100 ms timeout (more than enough) you're done in  less
> than 2 seconds worst case.

Yup, but it makes more sense to talk about RTT's than seconds :)

> You mean, rely just on ICMP and not announce a bigger MTU in RAs?

yeah pretty much.

> otherwise 64k jumbo-enabled network, because all those probes would  eat
> up my bandwidth even though I can't successfully receive them.

You'd get one probe when you talked to a new host, which would
immediately fail.

> ones, especially with these large packets.

Indeed.  The problem with probing for large MTU's is that the probes
themselves are large :)

> Not sure if we want to do this check for every packet. 

It's a simple test, and besides, you don't have to do it for every every
packet, just the packets that matter :)

> Also, an

Yes, this is a problem, but then again, presumably they can forge the NS
reply anyway.  All that this means is that you need to sign your packets.

> With an MTU option in it. 

yeah that's probably wise, although not necessary.  You could probe
always and pray.

> And why not NS?

Because when A talks to B, you want A to do the MTU discovery and for B
to "learn" the MTU too, but you don't want both sending MTU probes, only
one of them needs to do so.

> If all of those switches announce their MTU, we're in business.

If we're making fanciful wishes, can I have a million dollars? and a pony?

While having switches announce MTU's is possible, I don't think you'll
get switch manufacturers to agree, and even if they do the old and/or
cheap switches won't announce it therefore leaving you to have to probe
anyway.  and even if all the switches announce their MTU, how do you
know which subset of switches your going through to get to the other end?

> information because we'll find out ourselves.

Exactly.

> If we don't do an MTU search and the switches don't announce their  MTU,
> you're probably not going to use jumboframes on such a network...

And you fall back to 1280 (or some other link defined minimum such as 1500).

> include the MTU there. 

Plenty of room for a "1500, small,medium or large jumbo frames" tho :)

> Gigabit does have some in-band stuff like flow
> control, maybe that can be reused. 

PAUSE frames yeah, not particularly useful but perhaps doable :)

> But you always run the risk that a

Yup.  You want to probe to find out.  I'm beginning to think you want to
probe Point to point links too to discover if they have any weird
limitations that they aren't announcing.

---

From: Mark Smith <[email-masked]>
Date: Thu, 28 Jul 2005 15:06:54 +0930

Hi Iljitsch, Perry,

I'll admit I haven't been fully following your thread of discussion,
I've been a bit busy on some other things (another reason why I like the
simple solution - less thinking :-) )

Something to keep in mind for your solution. TCP announces the maximum
segment size it can receive in the SYN or SYN/ACK segments, and I'm
pretty sure these days that is derived directly from the (current) MTU
value of the interface it is going to use at the time of the 3-way
handshake. I'm pretty sure that irrespective of if PMTUD (or another
local link method) discovers a larger possible PMTU value at a later
time during the TCP connection, existing TCP connections will not be
able to "burst" up to it.

Assuming that you're not planning to modify TCP as well, this means that
the knowledge of the "best" possible MTU that TCP can use must be known
at the time of the initial 3-way handshake. This limitation possibly
applies to other unicast or semi-unicast transport layer protocols too,
such as SCTP.

Thinking about these solutions another way, all they're really
attempting to do is to take the TCP-specific MSS mechanism, generalise
it so it can be used for all transport layer unicast connections, and
moving the mechanism to the network layer.

Regards,
Mark.