Subject: [RAM] Tunnelling & GigE jumbo frame size

From: RJ Atkinson <[email-masked]>
Date: Wed, 12 Sep 2007 10:24:52 -0400

Dino's notes all sound correct, with one exception.

The exception is a detail and does not alter his main point,
which is that by using Jumbo Ethernet, tunnelling
of 1518 byte Ethernet frames ought not create any
fragmentation issues.  However, just to be crystal clear
to anyone who isn't tracking Ethernet closely, here
is the summary analysis.

Correction:
	Just as "4K" MTU really means 4470 bytes, Jumbo Ethernet
	sizes of "9K" really mean an IP MTU of 9180 bytes.  This
	means the link MTU for Jumbo Ethernet is really:
            (9180 + sizeof(Ethernet header) + sizeof(VLAN tag))

	Implementers who want to be more widely deployable will
	have a "9K" link MTU that allows for at least 2 VLAN tags
	(considering the widespread use of Q-in-Q encapsulation)
	or 1 VLAN tag plus 2 Ethernet MAC headers (for MAC-in-MAC
	encapsulation).   Both Q-in-Q and MAC-in-MAC encapsulations
	are commonly used in Metro Service Provider deployments
	of Ethernet (e.g. in Japan, Korea, Sweden, and in the US).

History:
	The Jumbo Ethernet 9K frame size was demanded by Ethernet
	users who were transitioning away from ATM/AAL5.  So they
	wanted Jumbo Ethernet to support existing IP packet lengths
	for IP/ATM AAL5 [see RFC-1626] *without* any risk of IP
	fragmentation.

Caveats:
	Jumbo Ethernet is explicitly not conformant to IEEE 802.3
	standards.  So all implementations that I'm aware of have
	a default MTU (i.e. out of the box) that is IEEE 802.3
	compliant.  Normally, users must explicitly configure the
	larger MTU size on each interface before it is enabled.

	1 GigE standards permit deployment with CSMA/CD, although
	I am not personally aware of any such deployments.  A GigE
	CSMA/CD deployment is likely not to work properly with
	an Ethernet frame size above that permitted by IEEE 802.3
	standards.  Switched Ethernet should be fine, however,
	and all actual 1 GigE deployments that I am aware of are
	switched and do not use CSMA/CD.

Yours,

Ran
[email-masked]

_______________________________________________
RAM mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ram

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 12 Sep 2007 18:42:06 +0200

> Dino's notes all sound correct, with one exception.

I'd like to support Pekka's point about internet exchanges, though.

> to anyone who isn't tracking Ethernet closely, here

Please note that ethernet frame size / MTU may or may not include the  
Ethernet_II header (which doesn't conform to IEEE 802.3) and if it  
does, it may or may not include the frame check sequence. I'll use  
the IP MTU excluding ethernet headers unless otherwise specified.

>            (9180 + sizeof(Ethernet header) + sizeof(VLAN tag))

I think most people take it to mean 9000 bytes. This is the only real  
jumboframe IP MTU size I've seen implemented in hosts. However,  
routers and especially switches come with many different maximum  
frame sizes. I've looked over some product literature and compiled  
the following list: 1508, 1530, 1536, 1546, 1998, 2000, 2018, 4464,  
4470, 8092, 8192, 9000, 9176, 9180, 9216, 17976, 64000 and 65280.

> 	switched and do not use CSMA/CD.

When gigabit ethernet was introduced the _minimum_ frame size (not IP  
MTU size) was (sort of) increased from 64 to 512 bytes when using  
CSMA/CD to make sure a frame would fill the entire collision domain  
even at the faster bit rate. However, the _maximum_ frame size was  
never increased because it would break old stuff. Lots of gigabit  
ethernet equipment supports larger frames, but few 10 or 100 Mbps  
implementations do. So having a larger frame size for GigE would make  
interoperation with older ethernet equipment a problem.

Although the issue comes up in the IEEE 802.3 working group every few  
years, no progress has been made here because the IEEE protocols  
simply don't have the hooks that make maximum frame size negotiation  
between ethernet systems possible. However, this is much easier at  
the IP level (especially IPv6), so I'm proposing that the IETF pick  
up the slack. See the discussion about my multi-mtu draft on the  
internet area list:

http://www1.ietf.org/mail-archive/web/int-area/current/index.html

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 12 Sep 2007 13:42:58 -0400

On  12 Sep 2007, at 12:42, Iljitsch van Beijnum wrote:
> I think most people take it to mean 9000 bytes.

I strongly disagree -- speaking as someone who spends a lot of time
talking with A LOT of different Ethernet end users around the world,
including users with deployments on all continents.  The origin
for the requirement came from super-computing sites with ATM
deployments, and they drove all the vendors in the same direction
(and vendors that didn't comply with their definition lost
potential business as a result; economics tends to win in such cases).

> This is the only real jumboframe IP MTU size I've seen implemented
> in hosts.

You should look around more broadly then.

I've seen the jumbo MTU that I outlined in multiple versions of UNIX
and in other kinds of systems.  Moreover, the jumbo MTU is primarily
deployed by service providers and super-computer centers, and is
not often used by ordinary end systems (e.g. one's home computer).

I'm not current with MTU support in current Windows, but earlier
versions of Windows did support more than 9000 bytes on Gigabit
Ethernet.

> However, routers and especially switches come with many different  
> maximum frame sizes. I've looked over some product literature and  
> compiled the following list: 1508, 1530, 1536, 1546, 1998, 2000,  
> 2018, 4464, 4470, 8092, 8192, 9000, 9176, 9180, 9216, 17976, 64000  
> and 65280.

As there is no standard, the maximum for a particular device will vary.
Most networking devices make this parameter *configurable*, below
some maximum value.  Varying maxima was not the question on the table,
however.

The question on the table is *narrowly* "what is meant by 9K MTU ?".
I stand by the contents of my original note.

Ran

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 12 Sep 2007 20:13:24 +0200

> I strongly disagree [...]

> You should look around more broadly then.

(Oh wait: I once saw a 10 Gbit ethernet card in a couple of PowerMac  
G5s that did 16384 bytes.)

> not often used by ordinary end systems (e.g. one's home computer).

Let me make several points at once. This is my computer at home, a  
Mac laptop:

$ sudo ifconfig en0 mtu 9180
Password:
$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         supported media: autoselect 10baseT/UTP <half-duplex>  
10baseT/UTP <full-duplex> 10baseT/UTP <full-duplex,hw-loopback>  
10baseT/UTP <full-duplex,flow-control> 100baseTX <half-duplex>  
100baseTX <full-duplex> 100baseTX <full-duplex,hw-loopback> 100baseTX  
<full-duplex,flow-control> 1000baseT <full-duplex> 1000baseT <full- 
duplex,hw-loopback> 1000baseT <full-duplex,flow-control> none
$ sudo ifconfig en0 mtu 9000
$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 9000
$ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

In other words: it can do 9000 bytes but not 9180 and the default MTU  
on the loopback interface is 16834 so obviously 9000 is not a maximum  
imposed by the OS. And apparantly, it can't do half duplex for  
gigabit ethernet.

> I stand by the contents of my original note.

Well obviously we shouldn't be saying "9k MTU" because it is too  
imprecise. Also, it's irrelevant unless we're trying to come up with  
a unified MTU size, which is something I am very much against, as it  
doesn't solve the problem but only moves the goalposts. We need to  
move to a situation where the capability to use larger packets is  
used where it exists and to the degree that it exists.

If we can increase the average packet size used on the internet by  
transmitting the same amount of data in fewer packets and routers/ 
switches are built such that they mostly use power when they are  
actually doing work, this will help with the power/heat issue.

But for the purposes of a jack up solution we just need a few dozen  
extra bytes to accommodate the new headers without triggering the  
breakage that ensues when 1500-byte packets can't be forwarded  
without fragmentation.

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 12 Sep 2007 11:22:00 -0700

> imprecise. Also, it's irrelevant unless we're trying to come up  
> with a unified MTU size, which is something I am very much against,  
> as it doesn't solve the problem but only moves the goalposts. We  
> need to move to a situation where the capability to use larger  
> packets is used where it exists and to the degree that it exists.

I will change the reference in the LISP spec from "jumbo frames of  
9000 bytes" to "jumbo frames of 9180 bytes". Will be reflected in the  
next ID update.

Dino

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 12 Sep 2007 14:34:45 -0400

On  12 Sep 2007, at 14:13, Iljitsch van Beijnum wrote:
> Let me make several points at once. This is my computer at home,
> a Mac laptop:

(A bunch of irrelevant stuff deleted here.  What a Mac
can or can't do isn't representative of anything beyond Apple,
certainly not of the entire world.)

> We need to move to a situation where the capability to use larger

I know of a LOT of metro SPs and ISPs and IXs that already support
the 9180++ MTU that I outlined before -- in their operational network
right now.  I think NDAs do not permit me to enumerate the list
that I know about personally...it isn't a short list in any event.

> breakage that ensues when 1500-byte packets can't be forwarded  

If you believe the above, why have you been wasting everyone's time
just now ?

Ran

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 12 Sep 2007 21:35:39 +0200

> the 9180++ MTU that I outlined before

Yes, but the way IP currently works means that ALL equipment in an IP  
subnet must share the same MTU. This is not a problem in itself if  
you have a homogonous ISP network (but the fact that you need to  
enable the larger size by hand is even there) but that requirement  
makes it extremely hard to deploy larger packet sizes in many other  
environments.

>> But for the purposes of a jack up solution we just need a few  
>> dozen extra bytes to accommodate the new headers without  
>> triggering the breakage that ensues when 1500-byte packets can't  
>> be forwarded without fragmentation.

> If you believe the above, why have you been wasting everyone's time

Does anyone believe anything else??

I guess I should read the latest LISP draft as apparently there  
specific a jumboframe size is mentioned, and I have no idea for what  
purpose.

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 12 Sep 2007 12:47:58 -0700

> specific a jumboframe size is mentioned, and I have no idea for  
> what purpose.

To summarize the issue we are trying to put to rest:

o All router-to-router links on the downstream side of an ITR use  
either 4470 or
   9180 byte MTUs.
o We don't care about the links inside a site because a packet is not  
LISP
   encapsulated there (in the typical case of a CE deployment of LISP).
o Hosts can originate 1500 byte sized packets.
o ITRs and TE-ITRs can prepend IPv4 and/or IPv6 headers without  
worrying about
   doing PMTU discovery or fragmentation.

So even when ~80 bytes of IPv6 is used for encapsulation, we have  
enough breathing room to avoid fragmentation (in the event we have 2  
headers, one for purposes of Loc/ID separation and the other for ISP- 
based TE).

Dino

---

From: Gert Doering <[email-masked]>
Date: Wed, 12 Sep 2007 22:02:47 +0200

Hi,

On Wed, Sep 12, 2007 at 12:47:58PM -0700, Dino Farinacci wrote:
> either 4470 or 9180 byte MTUs.

Where is this assumption coming from?  As long as there are FastE links,
this is not going to happen.

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  122119

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 12 Sep 2007 16:07:41 -0400

On  12 Sep 2007, at 16:02, Gert Doering wrote:
> Where is this assumption coming from?
> As long as there are FastE links, this is not going to happen.

Are you trying to say that you currently have deployed
router-to-router links inside an ISP/IX with less than
4470 bytes of IP MTU ?

Btw, at least some 100baseTX/FX switches/routers can
support the 9180 IP MTU.  So it is not necessarily
the case that 100baseTX/FX implies a 1518 byte link MTU.

Ran

---

From: Gert Doering <[email-masked]>
Date: Wed, 12 Sep 2007 22:14:52 +0200

Hi,

> router-to-router links inside an ISP/IX with less than

We have no single ethernet link in our network that has a MTU *above* 1530
(there are some that could be cranked up, but there was no need yet)

I have a large number of links that have a MTU of *exactly* 1500 (because
there is equipment that cannot handle 1530).

All exchange points that we're connected to run the fabric at 1500 byte
MTU, because they have members that have equipment that cannot handle
more.  There are *some* IXPs that have two different LANs, one with
1500 and one with "Jumbo", but that's not very widespread yet.

So: the answer is "yes".

> the case that 100baseTX/FX implies a 1518 byte link MTU.

Switches are easy.  Routers are not.

What good is having a Jumbo-capable switch if there is one peer on the
shared fabric that can only receive 1500-byte (+ Ethernet header) packets?

Gert Doering
        -- NetMaster