Subject: [Isis-wg] More vendor data on IP/GigE MTU

From: Ran Atkinson <[email-masked]>
Date: Sat, 16 Jun 2001 09:49:02 -0400

A quick check reveals that GigE host adapters from Intel, 3COM,
and Broadcom all support a 9K MTU.  Further, widely used inexpensive
commodity chipsets, such as the Intel LXT1000 and Broadcom BCM-5701, 
are specifically designed to support the 9K IP/GigE MTU (confirmed
just now by reviewing those data sheets).

	WindowsNT, Windows2000, and WindowsXP all support the 9K MTU.
Many other server vendors also support the 9K MTU for IP/GigE.

	I'm unaware of (and unable to locate quickly) any 10/100 Ethernet
products that support anything other than the standard 1518 byte MTU.
Data on any specific 10/100 products that support an MTU larger than 
1518 (+ allowance for 802.1q VLAN tags && 802.1p precedence) would 
be interesting to me at least.

Cheers,

Ran
[email-masked]

_______________________________________________
Isis-wg mailing list  -  [email-masked]
http://external.juniper.net/mailman/listinfo/isis-wg

---

From: Tony Li <[email-masked]>
Date: Sat, 16 Jun 2001 14:42:28 -0700 (PDT)

Ran,

Apparently we're trying to solve different problems.  

I'm trying to eliminate the need to do high speed fragmentation in the
backbone of the Internet.  Having multiple MTU sizes is not a good thing
for anyone and it causes unnecessary inefficiencies for both the router and
the end host that has to do the reassembly.  

From that perspective, adding another MTU size to the Internet simply
increases the need for fragmentation.  Server sends 9180B packet to
router A.  Router A fragments to 4470B to get through a POS link.
Somewhere down the path, router B fragments down to 1500B to actually
deliver to the user over 10BaseT.  How did this help?

The rationale in RFC 1626 is flawed.  At the very least, modern NFS runs
over TCP, not UDP.  The rest of the RFC has no compelling rationale for a
larger MTU.  RFC 1626 does get one thing right, tho:

       Fragmentation of IP datagrams is known to be highly
       undesirable. [KM87] 

Do you no longer agree with this?

Tony

---

From: Jesper Skriver <[email-masked]>
Date: Sat, 16 Jun 2001 23:57:09 +0200

> router A.  Router A fragments to 4470B to get through a POS link.

pMTUd is what's used generally, so fragmentation by routers isn't
seen much in real life.

/Jesper

-- 
Jesper Skriver, jesper(at)skriver(dot)dk  -  CCIE #5456
Work:    Network manager   @ AS3292 (Tele Danmark DataNetworks)
Private: FreeBSD committer @ AS2109 (A much smaller network ;-)

One Unix to rule them all, One Resolver to find them,
One IP to bring them all and in the zone to bind them.
_______________________________________________
Isis-wg mailing list  -  [email-masked]
http://external.juniper.net/mailman/listinfo/isis-wg

---

From: Tony Li <[email-masked]>
Date: Sat, 16 Jun 2001 15:13:56 -0700 (PDT)

| pMTUd is what's used generally, so fragmentation by routers isn't
 | seen much in real life.

If that were true, then my life would be a GOOD bit easier.  Sigh.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 17 Jun 2001 05:00:33 -0700

Jesper,
 
> pMTUd is what's used generally, so fragmentation by routers isn't

Well I am not sure what is your definition of "real life", but at least
in mine the fragmentation is a nightmare of each customer deploying mpls
in his network. By adding 4 or 8 bytes to already maxed ethernet frame
you either must support jumbo frames or fragment. 

As a contrary I would say that MTU path discovery almost never works :).
In 90 % of cases I have seen this is due to misconfigured firewalls (or
as security team say: "configure very tightly" ;). The rest 10% of pmtud
reasons for failures can be found here:
http://www.cisco.com/warp/public/105/38.shtml

R.

---

From: Jesper Skriver <[email-masked]>
Date: Sun, 17 Jun 2001 14:17:18 +0200

> you either must support jumbo frames or fragment. 

Yes, you need to be able to support atleast 1508 bytes to avoid
fragmentation, but again allmost all modern operating systems do path MTU
discovery ...

> As a contrary I would say that MTU path discovery almost never works :).

As said most OS's have it enabled, and if it doesn't work, there
will be no connectivity, as the routers can't fragment the
packets, and yes I have seen the misconfigured packet filters and
firewall's, it's a pain, but saying that pMTUd almost never work,
is not a accurate statement.

And as most/all OS's have pMTUd enabled, the routers won't have to
do much fragmentation.

/Jesper

---

From: Tony Li <[email-masked]>
Date: Mon, 18 Jun 2001 02:43:19 -0700 (PDT)

|         Au contraire.  I fully agree with this.  It is precisely
 | for this reason that we should settle on 9180.  9180 is already
 | quite widely deployed and supported in the Gigabit Ethernet world.
 | Having a 9180 end-to-end MTU supported, which this draft can
 | greatly facilitate, will reduce current fragmentation of 9180
 | into smaller packets (for situations where PMTU is not in use
 | or has been broken somehow).  POS is has a configurable MTU.
 | FDDI is effectively dead, so we ought not limit ourselves to
 | the 4K MTU of FDDI.

Well, unfortunately, no one let all of the manufacturers of POS links in on
your little plan.  The default as shipped is 4470 and I don't know of a
single one that has the buffering to support 9180.

Also, 9180 is NOT necessary to support page flipping.  4470 would seem to
very nicely support a single 4k page.

I agree that NFS over UDP isn't dead yet.  But I sure don't think we should
be optimizing the entire Internet to support a protocol that we're trying
to kill off.

Tony

---

From: RJ Atkinson <[email-masked]>
Date: Sun, 17 Jun 2001 21:28:39 -0400

At 17:42 16/06/01, Tony Li wrote:
>for anyone and it causes unnecessary inefficiencies for both the router 
>and the end host that has to do the reassembly.  

        The above is a fine rationale for not supporting any form of
jumbogram and just using an end-to-end MTU of 1518 bytes.
Its not consistent with the draft, however.

>router A.  Router A fragments to 4470B to get through a POS link.

        Of course, POS links are configurable.  Many, even in Internet
core routers, happen to be configured for 1518 bytes.  See above.

>larger MTU.  

        Page flipping in servers is a significant performance advantage.
Further, per-packet overhead costs dominate per-byte costs in most
systems.  So having a larger MTU significantly improves the throughput
in practice.  This advantage is not changed by use of NFS/TCP vice 
NFS/UDP.  While  I'm a big fan of NFS/TCP, it has not (in practice) 
eliminated broad use of NFS/TCP.

>RFC 1626 does get one thing right, tho:
>

        Au contraire.  I fully agree with this.  It is precisely
for this reason that we should settle on 9180.  9180 is already
quite widely deployed and supported in the Gigabit Ethernet world.
Having a 9180 end-to-end MTU supported, which this draft can
greatly facilitate, will reduce current fragmentation of 9180
into smaller packets (for situations where PMTU is not in use
or has been broken somehow).  POS is has a configurable MTU.
FDDI is effectively dead, so we ought not limit ourselves to
the 4K MTU of FDDI.

Cheers,

Ran

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 18 Jun 2001 09:13:32 -0400

At 05:43 18/06/01, Tony Li wrote:
>Well, unfortunately, no one let all of the manufacturers of POS links 
>in on your little plan.  The default as shipped is 4470 and 
>I don't know of a single one that has the buffering to support 9180.

        I know of some POS interfaces that do have RTT ms of
buffering for a 9180 byte MTU.  I don't know if these are the
ones that you built or not, perhaps not given your comment.
Big fast routers with 9180 byte MTU on their GigE interfaces
do exist off the shelf from multiple vendors today.

>Also, 9180 is NOT necessary to support page flipping.  
>4470 would seem to very nicely support a single 4k page.

        True, but 8K pages are commonly used these days, not 4K,
so its of much less benefit than it was in the FDDI era.

>I agree that NFS over UDP isn't dead yet.  But I sure don't think 
>we should be optimizing the entire Internet to support a protocol 
>that we're trying to kill off.

        I didn't know you were on a crusade to kill off NFS.  Over
in other IETF WGs, there are efforts to improve NFS.  As noted
before, page flipping means that 9180 is the right size for
NFS/TCP as well.  Even if one isn't running NFS, the per-packet
overhead in most end systems means that the only way to get 1 Gbps
through a GigE interface is to support large jumbograms.  This
is why the de facto MTU for GigE jumbograms is 9180 bytes.

        More generally, *most* Gigabit Ethernet products that have
shipped in the past 3 years *already* support the 9180 MTU, including
big fast routers with GigE interfaces.  So you forgot to tell the rest 
of the world about your plan for 4470 to be the standard MTU.  This 
means that fragmentation is *already* inevitable on GigE links
(if PMTU is turned off or broken in some way).  My main goal is
to document widely deployed existing practice.  I happen to agree
with the reasons that such is the existing practice as well
(obvious to anyone reading this thread).

        As I noted at the outset, an Informational RFC is totally
non-binding (and the content of standards-track RFCs are pretty 
frequently ignored as well), so I don't see why you are getting
spun up.  

        I think your perspective and mine are pretty well established
at this point (and neither seems to have changed since the prior
conversation when RFC-1626 was an I-D :-).  It would be much more 
useful and productive (IMHO) to hear from other folks here on the list, 
to see if there is consensus one way or the other (perhaps we're both
confused :-).

Cheers,

Ran
[email-masked]

---

From: Russ White <[email-masked]>
Date: Tue, 19 Jun 2001 07:27:32 -0400 (EDT)

All of the pos links I've checked are 4470 mtu, so that's what
I'd go with. I don't see where the 9180 is gaining you much more,
at least until media that support this sort of mtu are widely
deployed. We can always push a change to this in the future, if
it would be advantageous to do so.

In the meantime, let's not beg for more fragmentation than we
already have.

:-)

Russ

> be optimizing the entire Internet to support a protocol that we're trying

_____________________________
[email-masked] <>< Grace Alone

---

From: Tony Li <[email-masked]>
Date: Tue, 19 Jun 2001 12:31:33 -0700 (PDT)

Ran,

 | >I agree that NFS over UDP isn't dead yet.  But I sure don't think 
 | >we should be optimizing the entire Internet to support a protocol 
 | >that we're trying to kill off.
 | 
 |         I didn't know you were on a crusade to kill off NFS.  

Only the UDP transport.

 | So you forgot to tell the rest 
 | of the world about your plan for 4470 to be the standard MTU.  

Yup, I assumed that this draft would have made it obvious.  Obviously not.
Oh well.  ;-) 

 |         As I noted at the outset, an Informational RFC is totally
 | non-binding (and the content of standards-track RFCs are pretty 
 | frequently ignored as well), so I don't see why you are getting
 | spun up.  

From the level of invective, you seem to be the one getting spun up.

As to it being Informational, we both know that that's a canard.  This is
what will get deployed and used.  We should agree, because it will become
reality.

 |         I think your perspective and mine are pretty well established
 | at this point (and neither seems to have changed since the prior
 | conversation when RFC-1626 was an I-D :-).  It would be much more 
 | useful and productive (IMHO) to hear from other folks here on the list, 
 | to see if there is consensus one way or the other (perhaps we're both
 | confused :-).

<WG co-chair>
I agree completely.  We've now heard from three parties, including
ourselves and not one of the authors of the draft.  This will be an
incredibly rough consensus given the importance of this.

I'll also point out that this difference of opinion also makes it obvious
that the draft DOES need to include some comment as to what the default MTU
should be.  We should not be moving forward without some decision.
</WG co-chair>

Tony

---

From: Tony Li <[email-masked]>
Date: Tue, 19 Jun 2001 12:48:50 -0700 (PDT)

| >I'm trying to eliminate the need to do high speed fragmentation in the
 | >backbone of the Internet.  Having multiple MTU sizes is not a good thing
 | >for anyone and it causes unnecessary inefficiencies for both the router 
 | >and the end host that has to do the reassembly.  
 | 
 |         The above is a fine rationale for not supporting any form of
 | jumbogram and just using an end-to-end MTU of 1518 bytes.
 | Its not consistent with the draft, however.

The draft can be changed.

 | >From that perspective, adding another MTU size to the Internet simply
 | >increases the need for fragmentation.  Server sends 9180B packet to
 | >router A.  Router A fragments to 4470B to get through a POS link.
 | 
 |         Of course, POS links are configurable.  Many, even in Internet
 | core routers, happen to be configured for 1518 bytes.  See above.

Not all POS links can be configured upwards.  Please pardon a short
digression into hardware design.

When designing a store-and-forward device, an interface will have a number
of buffers that it will receive frames into before passing them on for
further processing.  Typically, there will need to be at least two of those
buffers (double buffering) to sustain packet rate and most commonly three.
This gives one buffer for the incoming packet, one for the packet being
handed off into the guts of the system and another just to insure that we
don't get into a bad race condition when both packets are swapping roles at
the same time.

Such buffers are typically on-chip and thus are relatively expensive.  They
also need to be MTU sized.  The combination of these two factors generally
push designers to want a smaller MTU.  It makes the hardware more
efficient, which in turn makes the system more efficient.

Obviously, there are other hardware designs that are possible.  One can do
scatter-gather, for example.  However, that makes the hardware more
complicated, not less.  It seems to me that if we want our systems to have
the peak performance, we should be optimizing them to be efficient and
simple. 

 |         Page flipping in servers is a significant performance advantage.

I agree.  However, we can't use this argument alone to track the evolving
page sizes in processors.  Over the years, we've seen the page size
increase (wasn't the VAX 512B?) and we will not be able to change the MTU
of the Internet to match.  There will always be something bigger.

At the same time, we know that 1500B is inconvenient for everyone.  Even
having the MTU be half or a quarter of the page size would be better as the
driver could then still do page flipping at its external boundary.

 | It is precisely

Again, there are many POS links in the world which will not work at 9180,
so this is not a deployable solution.  You can come down.  We can't go up.
It's that simple.

Tony

---

From: Vijay Gill <[email-masked]>
Date: Tue, 19 Jun 2001 15:59:23 -0400

> Yup, I assumed that this draft would have made it obvious.  Obviously not.

Most of the backbone devices that go into networks that I have some
familiarity with are required to support at least 4470 MTU (+ whatever
overhead for labels, et al is needed). Reasons for this are varied but in
general tend to be something along the lines of datacenter to datacenter
traffic and OC-x connected customers with servers doing jumbos on gigabit
ethernet devices wanting to talk to other branch offices, and therefore
we'd like to avoid fragmentation if possible.

As an overall percentage of traffic, this is negligible, but thats what I
ask for from an operational perspective.  I don't care if someone supports
9k jumbos, but I do care that my pos and gige boxes can support 4470 MTU.

/vijay

---

From: Vijay Gill <[email-masked]>
Date: Tue, 19 Jun 2001 16:29:42 -0400

> FDDI is effectively dead, so we ought not limit ourselves to

Ran, for various reasons, including some I sent earlier to the list, from
an operational perspective, I've never seen any demand for > 4470+overhead
MTU. Therefore, any devices that go into networks that I've worked on have
been required to support an MTU of ~4470, and I do not see this changing
in the future. The amount of traffic at even 4470 is fairly small, but
there have been customers who have asked for a 4k MTU. I see no
operational customer based demand to go greater than 4k, and I suspect
this will be true for a while.

/vijay

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 20 Jun 2001 08:53:25 -0400

At 15:59 19/06/01, Vijay Gill wrote:
> least 4470 MTU (+ whatever

Vijay,

        I'm confused by your reasoning method.  As currently deployed, 
"jumbos on gigabit ethernet devices" have an MTU of 9180, not 4470.  
If your comment had said "jumbos on gig ethernet devices are uncommon", 
then the conclusion in favour of 4470 would have been a ton more obvious 
to me.  Saying that "jumbos on gigabit ethernet devices" and wanting
to avoid fragmentation are reasons for 9180 instead of 4470.
(Clarification welcome. :-)

        There is also the small matter that a number of exchange points 
(notably LINX, PAIX) support 9180 MTU Ethernet today.  I recognise
you're on the AboveNet side rather than the PAIX side, but any
comments on that ?

Ran

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 20 Jun 2001 09:02:05 -0400

>so this is not a deployable solution.  You can come down.  We can't go up.

        Having done a fair amount of hardware work lately, including
some on POS interfaces, I'll simply note that its entirely practical
and cost-effective to design a POS interface that can have 
variable amounts of memory buffering.  Some other vendors have 
shipped these, even if you have chosen not to do this.

        As to the GigE MTU, a number of implementations can't change 
MTU downwards easily without spinning hardware (this is one of the
business risks associated with very highly integrated devices; entirely
analagous to the business risk of not designing a POS interface to
be able to support enough buffer memory) and getting paying customers 
very very upset with us.  GigE vendors went to 9180 due to strong paying 
customer demand, not on some whim.  That noted, if folks only offer 
GigE frames with a 4470 MTU, then they obviously get passed through 
GigE devices just fine.  It is not realistic to expect that existing 
GigE devices will be doing the fragmentation on behalf of some 
downstream device that's out of step with the rest of the GigE world. 
Most hosts use software for generating packets, so the originating
hosts notionally could reconfigure their MTU down to 4470 on their
GigE interfaces.  No doubt there is some "registry" variable for this
in Windows, for example.

        It sure sounds like we had all better start trying to figure
out how to make PMTU work more reliably... :-)

Regards,

Ran

---

From: Vijay Gill <[email-masked]>
Date: Wed, 20 Jun 2001 15:31:38 -0400

On Wed, 20 Jun 2001, RJ Atkinson wrote:

Ran,

> (Clarification welcome. :-)

Apologies for the unclear wording. I really should have said 'all we
_cared_ about was ~4k mtu.' If it went higher, ok, we set it to 4470 and
moved on.  The point being that we didn't care about an mtu higher than
~4k, we only ensured that the edge-edge MTU in the network was at least
4470. Now this was the result of historical accident perhaps, but the only
MTU related questions (a few) customers asked was if we had a 4k mtu, and
we could answer in the affirmative to that. No one asked for a 9k that I
was aware of, so we did not bother.

>         There is also the small matter that a number of exchange
> points (notably LINX, PAIX) support 9180 MTU Ethernet today.  I
> recognise you're on the AboveNet side rather than the PAIX side, but
> any comments on that ?

I work for Metromedia, not AboveNet.  PAIX questions would be best
answered by Paul and company, but I do know that there has always been a
push for large MTU by those folks.

/vijay