Subject: Re: [RAM] Tunneling overheads and fragmentation

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 10 Sep 2007 15:03:50 +0200

[caught up, yay!]

On 21-jul-2007, at 16:40, Robin Whittle wrote:

> Even then, I fear that in order to preserve both reachability and
> efficiency (and any reachability problems which arise from
> fragmentation), that hosts in all networks, including non-upgraded
> networks, will need to adopt a somewhat lower MTU setting - for
> all the packets they send.

That would assume that there is something that doesn't support the  
"regular" MTU (in theory, there is no such thing, in practice: 1500  
bytes) between the place the packets are encapsulated and the place  
the packets are decapsulated.

I'm still operating under the assumption that both those places are  
in ISP networks. Now obviously there are lots of places in ISP  
networks that only support 1500-byte packets, but what would be a  
better decision here: push out a reduced packet size EVERYWHERE which  
we probably won't be able to raise any time soon, or require ISPs to  
either:

1. Implement path MTU discovery correctly, or:
2. Make sure that all encapsulated packets travel over paths that  
support at least 1500 byte + encapsulation sized packets

> I wonder to what extent every possible application respects the
> operating system's MTU.

The two are unrelated. Applications talk to transport protocols. The  
most popular one, TCP, breaks large chunks of data into smaller  
segments and coalesces multiple small chunks into larger segments.  
UDP simply adds its header and hands over the packet to the IP layer.  
The IP layer will fragment packets that are too large to be sent out  
over the interface of choice and/or the packet's destination. The  
only time there are problems (except of course from firewalls that  
don't like fragmentation) is when the transport protocol or  
application doesn't want fragmentation (DF=1) but the packet is  
larger than the interface MTU. Not sure what happens then, except  
that there is no way that a packet larger than the interface MTU is  
sent in one piece.

> I wonder if there are any widely
> used applications, such as games, P2P programs etc. which are
> hard-coded to assume a certain MTU which is close to, or right at,
> the limit of what can safely be sent across most of the Net.

Mostly video streaming, although that's all quickly moving to TCP  
these days. A common packet size here is 1450 bytes but I think  
that's excluding 28 bytes IPv4 + UDP so the packets are really 1478  
bytes.

I also remember seeing fragmented TCP traffic coming from DSL users.  
Obviously some link didn't do 1500 bytes in the middle but rather  
than use PMTUD the devices involved fragmented. I think DF wasn't set  
in this case, but it could have been cleared by a router or other box  
somewhere along the way. (I once implemented that myself when I got  
dial-up service delivered over a tunnel that only supported a 576- 
byte MTU.)

> Perhaps, in an optimistic scenario, recognising that some packets
> are to be encapsulated by ITRs, an ISP network could set up its
> DHCP system or whatever it is which gives DSL modems their
> parameters, to reduce the MTU and MSS settings sufficiently that
> the ITR-applied encapsulation still results in packets which will
> not be fragmented in most transit and border routers.

There are several MTU-related DHCP options but I'm not sure how  
widely they are supported. Note by the way that ATM which is used for  
both ADSL and DOCSIS (cable) supports MTUs much larger than 1500 bytes.

By the way, I'm currently working on this:

http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-01.txt

It doesn't directly address this issue but it allows for systems with  
different MTUs to coexist on the same subnet so it becomes a lot  
easier to deploy jumboframes.

> This really needs to be done for all hosts in all networks - not
> just hosts in networks which have been upgraded with ITRs and ETRs
> etc.

Oh joy.

Doing path MTU discovery is probably easier, and note that if one  
host in a TCP session has a reduced MTU, it will let the other know  
during the three-way handshake so the unencumbered host won't send  
packets that are too large.

> Even if the host had its own ITFH
> (Ingress Tunnel Function in Host) function, I don't see how the
> operating system could tell application programs that there is one
> MTU and MSS setting for packets going to some addresses and
> another setting for packets going to other addresses.

Not a problem, path MTU discovery already does this today.

However, I see issues with hosts doing their own en/decapsulation:  
that way, the locators are exposed to hosts and they are at risk for  
becoming just as unrenumberable as IP addresses today.

The only way to make sure that you can renumber easily is if there  
are no firewall rules looking at locators. And the only way that will  
happen is if the relationship between location and identity is strong  
enough that spoofing it is not a viable attack vector. Concretely:  
today the routing system is unspoofable enough that people filter on  
IP addresses. The routing system is run by service providers who  
generally aren't in the business of attacking people. But if a  
locator mapping system is also open to end-users, it may be possible  
for attacks to use this path and people won't be happy to filter on  
just identifiers, just like they aren't happy to filter on just DNS  
names today. So either the ISPs must run it or there must be a heavy  
layer of magic security dust.

> UDP encapsulation, as used by LISP and I think eFIT-APT, involves
> 20 bytes for the IP header, 8 bytes for the UDP header and some
> number of bytes, such as 4, for extra stuff

What exactly was the reason for UDP encapsulation again? I think Dino  
said something about load balancing and firewalling. I'm not buying  
that: you can load balance on the destination IP address and anyone  
running firewalls in the middle of the routing system is best served  
with a single deny any any rule.

> This is where IPv6's long addresses and headers become really
> ugly. There would be 40 bytes for IP-in-IP and 52 for basic UDP
> encapsulation.

We really need bigger packets to offset the ever-increasing overhead.

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 11 Sep 2007 23:01:44 +0200

[back on RAM because I believe that's where the LISP folks hang out  
and this is relevant for them, besides, people may think I'm a  
researcher...]

On 11-sep-2007, at 22:05, Brian E Carpenter wrote:

>>> I'm still operating under the assumption that both those places  
>>> are in
>>> ISP networks. Now obviously there are lots of places in ISP networks
>>> that only support 1500-byte packets,

> Can somebody provide evidence for this statement?

Minira#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Minira(config)#interface fastEthernet 1/0
Minira(config-if)#ip mtu ?
   <68-1500>  MTU (bytes)

Granted, this is a fairly old box in a small network, but I don't see  
anyone seriously claiming that ALL ISP networks support packets  
larger than 1500 bytes on ALL their internal links (and also on inter- 
ISP links).

That said, it may very well be feasible to engineer a "jack up"  
solution such that the encapsulated = larger packets only see parts  
of networks that support more than 1500 bytes with a level of effort  
that is within reason. (Let me put it this way: what's easier to  
support: a million prefix in the routing table or a 1600 byte MTU?)

>>> but what would be a better decision
>>> here: push out a reduced packet size EVERYWHERE which we probably  
>>> won't
>>> be able to raise any time soon, or require ISPs to either:

>> All these options look impossible or ugly to me.

> The first one has been eluding us for years, but the network
> still works. What's the evidence on actual deployment? (Also
> see below.)

Please note that this is about the network between *TRs, which is a  
much simpler universe than the host-to-host universe where weird  
OSes, NAT and firewalls get in the way. (Again, assuming that end- 
users don't get to run en/decapsulation boxes themselves.)

> The second one sounds like something that is in the ISPs'
> enlightened self interest, in which case it will happen.

Right. However, that would probably still be a deployment problem  
because we may have to wait for upgrades to happen. Elsewhere, I was  
more or less dragged into a discussion about tunnel MTU issues.  
Believe me when I tell you that  this is a minefield. However, the  
good thing with a LISP-like solution is that we get to design  
everything on all ends (ITR, ETR, mapping) so it's not too hard to  
implement fragmentation at the encapsulation layer. The issue with  
IPv4 encapsulation is that the ID space is too small to support  
decent packet rates. With something like LISP, we can implement an ID  
field there and severely tighten the reassembly window (maybe even go  
so far that fragments from one source must arrive in-order) so this  
works much better. The mapping system could distribute MTU/MRU  
information if that's helpful.

[...]

> Hence RFC 4821, which does need to get deployed.

RFC 4821 (PMTUD without ICMP) is great for TCP, but it's not  
reasonably implementable for most UDP-based protocols/applications.  
It also suffers from the problem that you don't know if your  
corresponent supports it so it requires a leap of faith that things  
will work out.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Sep 2007 14:41:31 -0700

> Granted, this is a fairly old box in a small network, but I don't  
> see anyone seriously claiming that ALL ISP networks support packets  
> larger than 1500 bytes on ALL their internal links (and also on  
> inter-ISP links).

I did a survey about a month ago and it is true. I will yield to  
those folks who responded to me to protect their privacy. ;-)

But the main gist was:

o We are going to 9K MTUs on all our internal links.
o Where we don't have 9K MTUs, we use 4470.
o Virtually no one runs ISP links at 1500.

Dino

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 11 Sep 2007 23:49:55 +0200

On 11-sep-2007, at 23:41, Dino Farinacci wrote:

>> see anyone seriously claiming that ALL ISP networks support  
>> packets larger than 1500 bytes on ALL their internal links (and  
>> also on inter-ISP links).

> o Virtually no one runs ISP links at 1500.

That's good news, because then we can apparently treat anyone using  
1500 as an exception that can be dealt with operationally rather than  
something that must be addressed in the protocol.

But what about inter-ISP links? I'm assuming this isn't a big issue  
for high capacity private peering, but here in Europe a lot of  
peering happens over exchanges, which obviously use equipment that  
can easily handle larger packets, but so many people are on a big fat  
shared subnet that it's a given at someone will bring down the lowest  
common denominator to 1500.

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Sep 2007 14:53:50 -0700

> o Virtually no one runs ISP links at 1500.

And hence why I put this paragraph (the second one below) in section  
5 of draft-farinacci-lisp-03.txt to reflect Brian Carpenter's comment  
on the subject:

-----

    Since additional tunnel headers are prepended, the packet becomes
    larger and in theory can exceed the MTU of any link traversed from
    the ITR to the ETR.  It is recommended, in IPv4 that packets do not
    get fragmented as they are encapsulated by the ITR.  Instead, the
    packet is dropped and an ICMP Too Big message is returned to the
    source.

    In practice, this is not really a problem.  Hosts typically do not
    originate IP packets larger than 1500 bytes.  And second, a survey
    has been taken (from a list of ISPs, see Acknowledgement section)
    where nearly all ISP link MTUs are either 4470 bytes or support
    Ethernet jumbo frames of 9000 bytes.  Therefore, we don't anticipate
    any problems with prepending additional headers.

-----

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 11 Sep 2007 14:54:46 -0700

> can easily handle larger packets, but so many people are on a big  
> fat shared subnet that it's a given at someone will bring down the  
> lowest common denominator to 1500.

All GigE and higher. We are okay.

Dino

---

From: Pekka Savola <[email-masked]>
Date: Wed, 12 Sep 2007 09:59:16 +0300 (EEST)

On Tue, 11 Sep 2007, Dino Farinacci wrote:
>> But what about inter-ISP links? I'm assuming this isn't a big issue for 
>> high capacity private peering, but here in Europe a lot of peering happens 
>> over exchanges, which obviously use equipment that can easily handle larger 
>> packets, but so many people are on a big fat shared subnet that it's a 
>> given at someone will bring down the lowest common denominator to 1500.
>

While it may be technically OK, it's nowhere near operationally 
simple, whether or not you count that as "OK".

Many/most inter-ISP links are through GE-based IXes.  I believe most 
of those use a common VLAN for all the peers.  So in order to upgrade 
from MTU 1500, all the ISPs connecting to the VLAN need to update 
basically at once to MTU 9000 or big packets get blackholed to 
non-updated peers.  Another alternative is having a separate "big-MTU 
VLAN", but moving operators from one VLAN to another introduces IP 
address changes, resulting in BGP flaps and rather big operational 
cost.  I believe these big-MTU VLANs haven't been very popular to date 
due to lack of demand (= you'll need to connect to both VLANs in any 
case, and there's rather little benefit _to date_ of supporting 1500+ 
inter-ISP MTUs).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Gert Doering <[email-masked]>
Date: Wed, 12 Sep 2007 22:08:37 +0200

Hi,

I seem to have missed that e-mail.  Sorry for replying not in sequence.

> >those folks who responded to me to protect their privacy. ;-)

We haven't been asked in this survey - and our data looks quite different.

> >o Virtually no one runs ISP links at 1500.

In nearly all locations we have devices that still have Fast Ethernet 
links (because upgrading to GigE is expensive and the amount of
packes flowing through the link doesn't require it).

While we don't run 1500 on links where MPLS is used, most Cisco gear
in use cannot go over 1520...1530 on FastEthernet ports.  This limits
the GigE machines to 1530 as well, as things really break (today) if 
you share a layer2 segment between machines with different MTUs.

So there is no way our network could run on a MTU of 4470 or even 
higher any time soon.

[..]
>    any problems with prepending additional headers.

This is handwaving, and not a good basis for research work.

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  122119

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Gert Doering <[email-masked]>
Date: Wed, 12 Sep 2007 22:11:15 +0200

Hi,

> All GigE and higher. We are okay.

You might be.  We're not.  

At the DECIX, currently the 3rd biggest exchange in Europe, about one 
third of the members (all sharing a common L2 network!) are still 
connected with 100 Mbit/s.

Which means "1500 for all of them".

Gert Doering
        -- NetMaster

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 12 Sep 2007 18:58:18 -0700

> So there is no way our network could run on a MTU of 4470 or even

Then you don't deploy ITRs or you deploy fragmenting-ITRs.

> This is handwaving, and not a good basis for research work.

You are right, we are trying to make a decision and are doing  
engineering now past the research stage.   ;-)

Dino