Subject: [RRG] MTU, jumboframes, ITR & ETR placement, ITR function in hosts

From: Robin Whittle <[email-masked]>
Date: Sat, 24 Nov 2007 13:58:03 +1100

Hi Iljitsch,

Responding to what you wrote in:

  PMTUD, Sprite & IPTM; Outer src-addr = sending host's addr
  http://psg.com/lists/rrg/2007/msg00599.html

and changing the subject line again . . .

I would be delighted if you can show me it is practical to insist
that ITRs and ETRs be installed only in places with significantly
greater than 1500 byte MTU to the rest of the Net.  As far as I
know, it is impractical, due to the widespread use of 100Mbps links
in many places.

The quotes below from the RAM discussion on this, especially those
from Gert Doering, seem to indicate that at present, there is
sufficiently widespread use of 100Mbps Fast Ethernet, to drag down
the MTU of many Internet exchange networks to 1500.

I don't discount what you are saying.  Maybe by the time an ITR-ETR
scheme is introduced, it will be practical to insist on a minimum
standard well above 1500.  I hope some other people can contribute
to this discussion.

However, while it would be possible to configure an ITR in some way
to tell it that it has a >>1500 byte MTU to the "core of the Net",
this doesn't help much, since it can't know for sure that every ETR
it needs to send packets to has a similarly high MTU.  I think we
need to have great flexibility in where ETRs are located, so I can't
imagine it being practical to assume all ETRs have an MTU above 1500.

Also, I think there are many benefits in having a caching ITR in the
sending host, as I discuss below.

Some messages on the RAM list concerning this are:

  http://www1.ietf.org/mail-archive/web/ram/current/msg01733.html
  Gert Doering:

     As of today, too many links and routers still have issues
     handling more than 1500 bytes of IP packets (like "most
     FastEthernet interfaces on Cisco routers", which is still
     considered "high bandwidth" over here).

     > What about routers in end-user networks, or CE routers of
     > providers?

     Even worse, due to DSL encapsulation you already end up with
     an IP MTU of 1492 bytes or less.

  http://www1.ietf.org/mail-archive/web/ram/current/msg01847.html
  http://www1.ietf.org/mail-archive/web/ram/current/msg01849.html
  Iljitsch van Beijnum:

     I don't see anyone seriously claiming that ALL ISP networks
     support packets larger than 1500 bytes on ALL their internal
     links (and also on inter-ISP links).

     Believe me when I tell you that this is a minefield. However,
     the good thing with a LISP-like solution is that we get to
     design everything on all ends (ITR, ETR, mapping) so it's not
     too hard to implement fragmentation at the encapsulation layer.

     But what about inter-ISP links? I'm assuming this isn't a big
     issue for high capacity private peering, but here in Europe a
     lot of peering happens over exchanges, which obviously use
     equipment that can easily handle larger packets, but so many
     people are on a big fat shared subnet that it's a given at
     someone will bring down the lowest common denominator to 1500.

  http://www1.ietf.org/mail-archive/web/ram/current/msg01854.html
  Iljitsch van Beijnum:

     I think most people take it {jumboframes} to mean 9000 bytes.
     This is the only real jumboframe IP MTU size I've seen
     implemented in hosts. However, routers and especially switches
     come with many different maximum frame sizes. I've looked over
     some product literature and compiled the following list: 1508,
     1530, 1536, 1546, 1998, 2000, 2018, 4464, 4470, 8092, 8192,
     9000, 9176, 9180, 9216, 17976, 64000 and 65280.

     (More interesting stuff on the innards of Ethernet.)

  http://www1.ietf.org/mail-archive/web/ram/current/msg01861.html
  Gert Doering:

     Dino Farinacci wrote:
     >> To summarize the issue we are trying to put to rest:
     >>
     >> o All router-to-router links on the downstream side of
     >>   an ITR use  either 4470 or 9180 byte MTUs.

     Where is this assumption coming from?  As long as there are
     FastE links, this is not going to happen.

  http://www1.ietf.org/mail-archive/web/ram/current/msg01862.html
  RJ Atkinson:

     Btw, at least some 100baseTX/FX switches/routers can
     support the 9180 IP MTU.  So it is not necessarily
     the case that 100baseTX/FX implies a 1518 byte link MTU.

  http://www1.ietf.org/mail-archive/web/ram/current/msg01863.html
  http://www1.ietf.org/mail-archive/web/ram/current/msg01864.html
  http://www1.ietf.org/mail-archive/web/ram/current/msg01865.html
  Gert Doering:

     In nearly all locations we have devices that still have Fast
     Ethernet links (because upgrading to GigE is expensive and the
     amount of packets flowing through the link doesn't require it).

     While we don't run 1500 on links where MPLS is used, most Cisco
     gear in use cannot go over 1520...1530 on FastEthernet ports.
     This limits the GigE machines to 1530 as well, as things really
     break (today) if you share a layer2 segment between machines
     with different MTUs.

     So there is no way our network could run on a MTU of 4470 or
     even higher any time soon.

     At the DECIX, currently the 3rd biggest exchange in Europe,
     about one third of the members (all sharing a common L2
     network!) are still connected with 100 Mbit/s.

     Which means "1500 for all of them".

     RJ Atkinson wrote (in part):
     > Are you trying to say that you currently have deployed
     > router-to-router links inside an ISP/IX with less than
     > 4470 bytes of IP MTU ?

     We have no single ethernet link in our network that has a MTU
     *above* 1530 (there are some that could be cranked up, but
     there was no need yet)

     I have a large number of links that have a MTU of *exactly*
     1500 (because there is equipment that cannot handle 1530).

     All exchange points that we're connected to run the fabric at
     1500 byte MTU, because they have members that have equipment
     that cannot handle more.  There are *some* IXPs that have two
     different LANs, one with 1500 and one with "Jumbo", but that's
     not very widespread yet.

     So: the answer is "yes".

     > Btw, at least some 100baseTX/FX switches/routers can
     > support the 9180 IP MTU.  So it is not necessarily
     > the case that 100baseTX/FX implies a 1518 byte link MTU.

     Switches are easy.  Routers are not.

     What good is having a Jumbo-capable switch if there is one peer
     on the shared fabric that can only receive 1500-byte (+
     Ethernet header) packets?

Regarding an ITR function in the sending host, this could be a
really good use of free CPU and RAM resources, saving upstream
dedicated ITRs from most of the work.  I don't see any special
security problems.  Can you point out any specific problems?  The
only ITR-ETR scheme which allows for an ITR function in a sending
host is Ivip:

http://tools.ietf.org/html/draft-whittle-ivip-arch-00#section-1.7.3

The ITR function in a sending host needs to have fast, efficient,
link to a nearby Query Server - so it would be a bad idea on dialup
or on a link to an ISP which didn't have its own full database Query
Server.

 - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 24 Nov 2007 23:34:59 +0100

> know, it is impractical, due to the widespread use of 100Mbps links

The practicality of that insistance isn't derived from wide  
availability of 1500+ byte MTUs in provider networks (although Dino  
claims this isn't an issue based on a survey he did) but rather from  
the trouble we can expect to see if we reduce the MTU of links across  
the core of the internet to below 1500 bytes.

> the MTU of many Internet exchange networks to 1500.

Obviously smaller ISPs will have 100 Mbps links in some places. The  
real question is: do the *TRs need to be behind those 100 Mbps links,  
or can they be placed in a more central part of the network where the  
requirement of a 1500+ byte MTU can reasonably be met?

You seem to want *TRs pretty much everywhere. Although I think there  
are security issues with that, I don't reject having them even in end- 
user networks out of hand, but that doesn't mean I accept the lowest  
common denominator in those networks. If they want to run an  
encapsulation/decapsulation device, they'll just have to upgrade their  
network to support the MTU that makes this possible.

> standard well above 1500.  I hope some other people can contribute

I don't expect the situation in this area to change significantly.  
Today, pretty much all 100 Mbps or slower stuff doesn't support  
jumboframes, while pretty much all 1000 Mbps or faster stuff does.  
(Sometimes this even goes for the 10/100 and 1000 ports on the same  
switch.) Expensive 100 Mbps equipment doesn't exist anymore, so I  
don't see a push for larger MTUs there.

> it needs to send packets to has a similarly high MTU.

This could be learned through the mapping service.

> sending host, as I discuss below.

In that case there are no problems, because obviously it's allowed to  
send packets smaller than the minimum maximum and PMTUD black holes  
aren't possible here because everything happens on the source host.

>     break (today) if you share a layer2 segment between machines

Yeah, someone should do something about that. Oh wait:

http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-01.txt

>     Which means "1500 for all of them".

Yes, this is an issue.

>     different LANs, one with 1500 and one with "Jumbo", but that's

That's because there is little value in configuring a larger  MTU  
today because in 99% of all paths through the internet there is at  
least one 1500-byte ethernet hop, so you're pretty much never going to  
see actual data packets flowing between end-users that are bigger than  
1500 bytes.

I'm pretty sure if LISP or something like it is deployed by the big  
guys in the US (which do all their interconnecting through private  
links AFAIK) the Europeans who use internet exchanges will spit and  
curse but make the necessary changes to the exchange setups after  
that. The alternative is endless handholding of customers with PMTUD  
problems rather than a one-time infrastructure change.

---

From: Robin Whittle <[email-masked]>
Date: Mon, 26 Nov 2007 01:08:29 +1100

Hi again Iljitsch!

Our correspondence continues:

>> I would be delighted if you can show me it is practical to
>> insist that ITRs and ETRs be installed only in places with
>> significantly greater than 1500 byte MTU to the rest of the
>> Net.  As far as I know, it is impractical, due to the
>> widespread use of 100Mbps links in many places.
> 
> availability of 1500+ byte MTUs in provider networks (although
> Dino claims this isn't an issue based on a survey he did) but
> rather from the trouble we can expect to see if we reduce the MTU
> of links across the core of the internet to below 1500 bytes.

This sounds like an argument between the immovable object and the
irresistible force.

Something doesn't become practical just because the only apparent
alternative seems unthinkable.

You later contemplate extending this to those end-user networks
which contain ITRs.  In practice, as the ITR-ETR scheme becomes
ubiquitous, this would be virtually all end-user networks.

The correspondence I quoted, from people who know much more about
this than me, convinced me that your desirable idea of upgrading to
MTU >> 1500 is not practical.

Perhaps it can be done, but I want ITRs and ETRs to be placed pretty
much everywhere, so I doubt this will be practical, since so many
edge networks run on 100Mbps Ethernet, which I understand typically
provides a Layer 3 MTU of 1500 bytes.

>> The quotes below from the RAM discussion on this, especially
>> those from Gert Doering, seem to indicate that at present,
>> there is sufficiently widespread use of 100Mbps Fast Ethernet,
>> to drag down the MTU of many Internet exchange networks to
>> 1500.
> 
> Obviously smaller ISPs will have 100 Mbps links in some places.
> The real question is: do the *TRs need to be behind those 100
> Mbps links, or can they be placed in a more central part of the
> network where the requirement of a 1500+ byte MTU can reasonably
> be met?
> 
> You seem to want *TRs pretty much everywhere. Although I think
> there are security issues with that, I don't reject having them
> even in end-user networks out of hand, but that doesn't mean I
> accept the lowest common denominator in those networks. If they
> want to run an encapsulation/decapsulation device, they'll just
> have to upgrade their network to support the MTU that makes this
> possible.

My proposal and Fred's is based on the assumption that we can't
avoid various things chopping the MTU below 1500 - so we proposed
something to cope with this which we think is reasonably (or to you
alarmingly) troublesome, but we still think it is practical and
better than the alternatives.

I suggest that Sprite and IPTM (or something like it) be developed
as best as possible.

Then, the costs and benefits of "Sprite as implemented for
ITR-ETR schemes" or IPTM can be compared with the costs and benefits
of what you propose.

>> I don't discount what you are saying.  Maybe by the time an
>> ITR-ETR scheme is introduced, it will be practical to insist on
>> a minimum standard well above 1500.  I hope some other people
>> can contribute to this discussion.
> 
> I don't expect the situation in this area to change
> significantly. Today, pretty much all 100 Mbps or slower stuff
> doesn't support jumboframes, while pretty much all 1000 Mbps or
> faster stuff does. (Sometimes this even goes for the 10/100 and
> 1000 ports on the same switch.) Expensive 100 Mbps equipment
> doesn't exist anymore, so I don't see a push for larger MTUs
> there.

I look forward to other people debating this.

>> However, while it would be possible to configure an ITR in some
>> way to tell it that it has a >>1500 byte MTU to the "core of
>> the Net", this doesn't help much, since it can't know for sure
>> that every ETR it needs to send packets to has a similarly high
>> MTU.
> 

ITR-ETR schemes other than Ivip have complex mapping systems and
their designers might contemplate adding something else to the
mapping data.  Ivip's system is very simple - just an ETR address
for a given micronet.  I don't want to add anything more than that.

If some ITRs want to participate in some collaborative system to
develop a cheatsheet of good PMTU values to try first, or to be
happy with as a maximum without pushing higher, that would be fine.
Perhaps that could be part of the whole ITR-ETR scheme.

It might work at the level of BGP advertised prefixes, rather than
specific ETRs.  Maybe the suggested PMTU value is the same for all
ETRs in that entire prefix.

The PMTU value doesn't need to be tied to every item of mapping
data, since, for instance, hundreds of micronets may currently be
mapped to one ETR.  There is only one "cheatsheet" PMTU value for
each ETR.

But this "cheatsheet" suggested value is of limited value, I think.
 There could be all sorts of PMTU limitations at the ITR end of the
tunnel, the ETR end or anywhere in the middle.  These may vary so
much (in the short term, as paths take the packets through different
routers with potential tunnels of their own?) that it is not
desirable to suggest that some single figure can be reliably used as
a default PMTU, or as a target to which a PMTUD system might first
try and be happy with.

>> Also, I think there are many benefits in having a caching ITR
>> in the sending host, as I discuss below.
> 
> In that case there are no problems, because obviously it's
> allowed to send packets smaller than the minimum maximum and
> PMTUD black holes aren't possible here because everything happens
> on the source host.

I haven't thought exactly how an ITFH might be implemented.  It
could be some additional functional block, inform the existing code
with the equivalent of a PTB message just as an external ITR would.
 Alternatively, it could be very tightly integrated with the code
which handles packetisation.

Previously, I recognised that an ITFH only makes sense in a given
location if it can quickly and reliably communicate with a nearby
Query Server.  However, this PMTU stuff involves the ITFH sending
hefty probe packets, which involves occasional, but perhaps
frequent, significant bursts of bandwidth.  This would typically be
more outgoing traffic than the queries.  Since DSL and cable modem
links have lousy upstream paths, this is a further reason not to put
an ITFH at the user end of such a link.  (I include in "ITFH" the
idea that the modem's NAT system perform the ITR functions on its
outgoing packets.)

I think an ITFH still makes a lot of sense to me for web servers in
a hosting farm, assuming they are not running flat out, to save the
hosting company's ITRs from handling most of the load.

> http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-01.txt

Sorry, I didn't look in detail at this, because it seems to be only
for IPv6.

>> Which means "1500 for all of them".

OK.

> today because in 99% of all paths through the internet there is
> at least one 1500-byte ethernet hop, so you're pretty much never
> going to see actual data packets flowing between end-users that
> are bigger than 1500 bytes.

This sounds familiar . . .  everyone (or almost everyone) has to do
an expensive upgrade before anyone derives any benefit.

We really need a Fairy Godmother to cast a spell which brings
forward some things in time (the benefits) so that the mechanisms
which provide the benefits can be created in the first place!

> I'm pretty sure if LISP or something like it is deployed by the
> big guys in the US (which do all their interconnecting through
> private links AFAIK) the Europeans who use internet exchanges
> will spit and curse but make the necessary changes to the
> exchange setups after that. The alternative is endless
> handholding of customers with PMTUD problems rather than a
> one-time infrastructure change.

I think we need to design a one-time infrastructure change resulting
in some new, flexible, address space which has certain limitations
which end-users can live with - but which does not suck, either
continually or in a flaky PMTUitis kind of way.

If some administrative arrangements could be used to cause all
relevant gear to be upgraded to gigabit Ethernet, I would be
delighted to forget about IPTM.  However, I think that would be a
cause of further delay for deploying Ivip - or whatever ITR-ETR
scheme is chosen - and/or for seriously restricting where ITRs and
ETRs could be located.

 - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 25 Nov 2007 18:09:22 +0100

> Hi again Iljitsch!

:-)

I'm not responding to your previous message because the way I see it,  
all those issues are sidestepped by simply mandating a minimum MTU of  
1500 in the *TR system.

> This sounds like an argument between the immovable object and the

Only if you assume that the requirement to have an MTU that will  
accommodate 1500 byte inner packets + encapsulation is equal to moving  
an immovable object or resisting an irresistable force.

> ubiquitous, this would be virtually all end-user networks.

Is it important for ITRs to become ubiquitous?

Is this more important than avoiding the situation where path MTU  
blackholes become very common? Like it or not, the de facto MTU of the  
internet is 1500 bytes. Go below that at your peril.

> MTU >> 1500 is not practical.

Depends on what exactly you want to upgrade. I fully agree this is a  
problem for people behind a DSL line using a sub $100 CPE. But I don't  
consider that an issue.

> much everywhere

If you want people with an MTU of 1500 to encapsulate, you have two  
choices:

1. Run PTMUD
2. Fragment

Unfortunately, you can't really do both at the same time unless your  
router keeps track of who it sent "too big" messages to recently. And  
I'm saying choice #1 isn't an option because of broken PMTUD behavior  
in way too many hosts connected to the internet. So that means  
fragmenting EVERY 1500-byte packet in the encapsulators and  
reassembling them in the decapsulators. That's an ongoing cost, while  
upgrading the mid-level potential runners of *TRs is a one-time cost.  
I'm not sure what you feel are the advantages of people below that  
level running *TRs as well (those are the people that aren't  
multihomed themselves) so please explain.

> Sorry, I didn't look in detail at this, because it seems to be only

No, that was version 00. There is support for IPv4 using jumbo ARPs in  
version 00 but it's less robust than IPv6 because IPv4 doesn't have  
any neighbor discovery.

> an expensive upgrade before anyone derives any benefit.

That is (somewhat) true for wide scale adoption of jumboframes on the  
internet. Supporting jumboframes on an exchange pretty much means that  
you have to split the existing peering LAN into two new ones: one for  
1500-byte packets and one for a larger size. This isn't a huge deal  
but it is of course somewhat annoying. (My draft wants to address this  
by allowing hosts with different maximum packet sizes on the same  
subnet.)

> scheme is chosen - and/or for seriously restricting where ITRs and

I think our respective positions are clear, I'd be interested to learn  
what others think.

---

From: Robin Whittle <[email-masked]>
Date: Mon, 26 Nov 2007 16:36:44 +1100

Hi again Iljitsch,

I hope others can join this discussion.

I think that ITRs and ETRs are going to be ubiquitous.  I can't see
how IPv6 (or any other imaginable replacement for IPv4) is going to
be widely enough adopted in the next decade or two to make ordinary
users happy without a public or "behind NAT" IPv4 address.  I think
an ITR-ETR scheme is the only way of handling increased numbers of
multihomed networks, and increased address utilization, without
causing a meltdown in the BGP routing system.

> internet is 1500 bytes. Go below that at your peril.

Clearly we can't have a major, systemic, increase in PMTU black-holes.

I hope some other folks can comment on your statement that PMTUD is
broken on way too many current hosts:

> I'm saying choice #1 isn't an option because of broken PMTUD behavior in
> way too many hosts connected to the internet. 

I assume you mean that the host doesn't respond to RFC 1191 "Packet
Too Big" (actually "Datagram Too Big") messages by resending the
message in smaller packets according to the Next-Hop MTU value in
that message.  I assume you are not referring to filters which block
these messages, either in the network of the sending host or in the
network nearer the destination host.

If these messages are dropped by filters in the sending host's
network, or anywhere between the sending host and the nearest ITR
(perhaps outside that network's ISP) then there is something wrong
with the filtering system and I think it should be fixed.

If host sending behaviour is genuinely broken in a generalised
manner, then then Fred's Sprite and my IPTM proposals would be
useless or close to it.

But if there are only occasional applications or operating systems
which don't do RFC 1191 PMTUD correctly, then wouldn't they have
trouble today, unless their default MTU was set somewhat below 1500?
 If there are just a few of these "broken hosts", I think it would
be better to weed them out than to pursue the only other apparent
alternatives:

1 - Insist on a radical upgrade to (in general) gigabit Ethernet
    gear in most networks (any network which connects to the Net) -
    in order that they can run ITRs.

2 - "Only" insist on a gigabit upgrade to all ISPs who want to run
    ITRs (we are intending they all do it pretty soon) and then
    generally accept that ITRs have to be in ISP networks and not
    in parts of end-user networks which don't have gigabit links
    to the ISP.

I suppose there is the prospect of re-engineering 100Mbps gear to
have a higher MTU, but I guess that means new silicon, with very
little profit from higher prices - so I assume this is not much of
an option.  Also, so many things have their Ethernet interface built
in, that there would need to be complete upgrades to many servers
and devices, not "just" plugging in faster or jumboframe-compatible
Ethernet cards into routers.

What about routers themselves?  Wouldn't the frame size be a
fundamental design decision?  In that case, I think you are
advocating a tremendous equipment upgrade everywhere, before the
ITR-ETR scheme (without Sprite or IPTM) could create benefits for
anyone.

If so, this would violate the requirements of incremental
deployment.  Sprite or IPTM should be able to meet those
requirements, by not requiring any massive hardware upgrades in ISP
and end-user networks all over the world.

> version 00 but it's less robust than IPv6 because IPv4 doesn't have any
> neighbor discovery.

OK - I will check it out.

On Jumboframes:

> 1500-byte packets and one for a larger size. This isn't a huge deal but
> it is of course somewhat annoying. (My draft wants to address this by
> allowing hosts with different maximum packet sizes on the same subnet.)
> 

Yes!

  - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 27 Nov 2007 11:34:12 +0100

On 26 nov 2007, at 6:36, Robin Whittle wrote:

>> because of broken PMTUD behavior in

> network nearer the destination host.

Actually I am. Or rather, I'm assuming this is the issue because I  
don't have access to the internals of broken PMTUD deployments. I  
would hope that the IP stacks aren't so broken that if the ICMP  
messages make it back in one piece it still doesn't work, some two  
decades after the RFC was published.

> with the filtering system and I think it should be fixed.

You and me both.

> trouble today

What you see is that if you are behind a link with a < 1500-byte MTU  
(but not directly attached to that link because then the TCP MSS  
option avoids trouble) that you can't reach certain destinations on  
the internet: sessions establish but then stall. In most cases, you,  
the client, will request something and the request is small but the  
reply is large. So this triggers PMTUD on the part of the remote  
website etc but they never adjust their packet sizes so nothing  
happens. If this happens with one site you could go after them and  
hopefully they'll fix it, but that doesn't work when it's tens or  
hundreds of thousands. The only option you have as a customer behind a  
PPPoE link or some such is for your CPE to adjust the TCP MSS so  
others never send you large packets in the first place. And because  
everyone does that, the people responsible for the problem continue  
their brokenness.

:-)

> an option.

Right, "100 Mbps gear with new silicon" would be 1000 Mpbs gear.

> Also, so many things have their Ethernet interface built

If it has gigabit, in most cases it will do jumboframes. My Dell  
FreeBSD machine:

# ifconfig em1 mtu 16115
ifconfig: ioctl (set mtu): Invalid argument
# ifconfig em1 mtu 16114
# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 16114

My Apple MacBook Pro laptop:

# ifconfig en0 mtu 9001
ifconfig: ioctl (set mtu): Invalid argument
# ifconfig en0 mtu 9000
# ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 9000

(Although my 4-year-old Powerbook also has gigabit ethernet but can  
only handle 1504.)

I'm not sure if there are any cable/DSL CPEs out there that support  
larger packets on both ends, though. My Cisco 826 can do 4470 on the  
DSL (ATM) side but it only has 10 Mbps ether and:

(config)#in e0
(config-if)#mtu 1504
% Interface Ethernet0 does not support user settable mtu.

> fundamental design decision?

Sure. But again, if it can do gigabit ethernet, it can probably do  
larger packets.

Don't forget that in the days before ethernet wiped out pretty much  
all other link technologies, there was lots of other stuff out there  
and it all supported more than 1500 bytes: packet over SONET, ATM,  
FDDI... So unlike switches, routers tend to be designed for some  
flexibility in this area.

---

From: Robin Whittle <[email-masked]>
Date: Wed, 28 Nov 2007 13:16:19 +1100

Hi Iljitsch,

In http://psg.com/lists/rrg/2007/msg00614.html I wrote that my
understanding of your message 611 was that so many hosts do not
respond properly to RFC 1191 "Packet Too Big" (PTB actually
"Datagram Too Big") messages for Path MTU Discovery to work with
Fred's Sprite proposal or my IPTM proposal.  Checking to see if this
is what you meant, or whether you meant the problem was due to
filters dropping the PTB messages, you replied that you meant the
latter (msg 628).

I think this latter problem should not prevent IPTM, or an ITR-ETR
implementation of Sprite from working well.

> What you see is that if you are behind a link with a < 1500-byte
> MTU (but not directly attached to that link because then the TCP
> MSS option avoids trouble) 

At least for TCP packets.  Maybe not for other protocols which
choose their own packet sizes and foolishly assume a 1500 byte MTU.

> that you can't reach certain
> destinations on the internet: sessions establish but then stall.
> In most cases, you, the client, will request something and the
> request is small but the reply is large. So this triggers PMTUD
> on the part of the remote website etc but they never adjust their
> packet sizes so nothing happens. If this happens with one site
> you could go after them and hopefully they'll fix it, but that
> doesn't work when it's tens or hundreds of thousands. The only 
> option you have as a customer behind a PPPoE link or some such is
> for your CPE to adjust the TCP MSS so others never send you large
> packets in the first place. And because everyone does that, the
> people responsible for the problem continue their brokenness.

I think it is not acceptable for such host adjustments to be
required in order to make an ITR-ETR scheme robust against PMTUD
black holes.

Here is a diagram depicting my understanding of what you describe:

Server                                                    Client

 SH-->--R1--->---R2--->---R3--->---R4--->---R5-->--R6-->--DH

Link 1       2        3        4        5       6      7

 Hosting co.     |   ISPA |        DFZ      | ISPB | DSL
 network         |        |                 |      | link

Let's say the reduced MTU is in link 7.  The  filter which drops the
ICMP PTB messages generated by the DH (Destination Host) is located in:

Case 1:  ISPB (R5 and/or R6).

Case 2:  ISPA (R2 and/or R3).

Case 3:  Hosting company (R1).

In Case 1, the client is going to have PMTUD troubles with all hosts
outside ISPB that try to send it packets 1500 bytes long, as many
will.  I assume Case 1 will not persist, because any ISP with
customers using MTU-challenged DSL links would be badly mistaken to
drop the ICMP PTB messages their customer's hosts generate.

In Case 3, the hosting company is shooting itself in the foot - but
as you describe, those who suffer are generally unable to figure out
what is wrong, and are too few and too widely scattered to provide
sufficient feedback for the hosting company to change their ways.

Case 2 is much the same, but I would think it daft for an ISP to
busy itself filtering out ICMP messages of any kind for one of its
presumably technically sophisticated customers.

Now I will consider these cases with an ITR-ETR scheme and either
Sprite or IPTM.  In both Sprite and IPTM, the ITR uses special probe
packets (not traffic packets) to determine the PMTU to the ETR.  The
ITPM ITR could receive PTB messages which result from probe packets
which are too large for some router en-route to the ETR, but the ITR
does not depend at all on PTB messages to determine the PMTU to the
ETR.  The ITR and the ETR have their own explicit protocol and the
test is what length probe packets are acknowledged by the ETR.

Once the ITR has reliably determined the PMTU to the ETR (or at
least stopped probing for higher values, being satisfied with the
relatively high value it has already proven to work) it can
communicate this to the Sending Host (SH) via an RFC 1191 PTB
message, assuming the SH sends a packet larger than this.  That
message contains a value such that the SH will send packets of a
maximum size, which when encapsulated, will just fit in the reliably
determined PMTU limit to the ETR.

Please see my recent message on how IPTM would behave and querying
Fred on how Sprite would behave:

  http://psg.com/lists/rrg/2007/msg00635.html

Anyway, back to Case 1, 2 and 3.  There ITR could be located
anywhere from the SH Sending Host itself to a router in the DFZ.

With all such ITR placements, Case 1 will generate persistent
trouble for customers, as it does today - so we would expect that
very few Case 1 filtering arrangements would persist.

For Case 2 and 3, what matters is whether the ICMP filter is between
the Sending Host and the ITR.  An ITR function in the host (Ivip
ITFH) fixes that problem!

In either Case 2 or 3, as long as the filtering is between the SH
and the ITR, there is going to be persistent trouble.  This is
because the ubiquitous use of the ITR-ETR scheme will cause far more
PMTU limitations than currently exist, and (assuming the ITR-ETR
scheme includes something like IPTM or Sprite) the filtering will
clobber the only way the ITR can tell the SH to send shorter packets
to cope with these restrictions.

Perhaps, in the early stages of ITR-ETR deployment, when only a
handful of Destination Hosts use the ITR-ETR-mapped addresses, there
may be such a low level of trouble that the errant party (the
hosting company or ISPA which is filtering out the ICMP packets)
doesn't get enough pressure to change their ways.

This would have the effect of making the ITR-ETR-mapped address
space suck - which we want to avoid like the plague.

Once the ITR-ETR scheme was widely used (I see no other option for
the Net), then I expect that the problems would become so persistent
that the errant party would ensure their filters allow the PTB
messages to go back to the SH.

  - Robin