Subject: Re: [Int-area] draft-van-beijnum-multi-mtu-00

From: John Heffner <[email-masked]>
Date: Sun, 26 Aug 2007 00:44:28 -0400

Comments on draft-van-beijnum-multi-mtu-00

First, I'd like to say that I agree with the goal of being able to run 
mixed-MTU subnets.  I'm not yet convinced this draft constitutes a full 
and practical solution to the problem.

I have some specific comments below divided by section.  A few general 
ones first:

One thing that might make the draft easier to read would be some 
discussion of how the pieces described in Section 4 fit together to form 
a solution to the problem presented.

I think the draft is a bit too focused on Ethernet.  For example, 
specifying 1500 byte minimum MTUs.  A standard in this area should apply 
equally to all link types.

The discussion in Section 3 is pretty good, but it might be a bit long 
and detailed for this doc.

4.1:

It's not clear what exactly the Allowed MTU and Off-link MTU come from, 
and how a router selects these values.  Is one of these the router's 
interface MRU?  Why do you need an off-link MTU?  As long as you don't 
send packets longer than the router's MRU, then PMTUD will take care of 
this.

The whole link speed thing makes me a bit uncomfortable.  You definitely 
want links with a much slower speed to use smaller MTUs, but it seems 
like the end hosts are able to make an appropriate decision.

4.3:

It bothers me having switches send this stuff.  It seems like an 
architectural violation to have a layer 2 device sending layer 3 
information.

Since this is broadcast, a single switch can pull down the MTU for an 
entire subnet.  Is that a good thing?

These are sent out every 5 minutes.  If a new device comes up, it might 
be operating for 5 minutes (or MAYbe 60 seconds) without this 
information?  And, it's not even required for a host to listen to it. 
I'm not sure I understand the justification for doing this at all.

4.4:

I think there's some promise in this area, but it's currently 
underspecified.  This section describes the MTU detection message, but 
doesn't really say what to do with it.  Do you need to get a reply 
before using a larger MTU?  If you don't get a reply at a given size, 
then what do you do?  Fall back to the minimum?  Probe at other sizes a 
la 4821?

You forbid sending MTU detection messages more often than once per 60 
seconds.  I don't see this as being practical (say, on a router 
interface) where you may need to send/forward packets to large numbers 
of hosts that may all need to be probed individually.

Ultimately I would like to see something done in this area.  I might 
even volunteer to help. :)

Lastly, I apologize for taking so long to send this out.  It was sitting 
nearly done, but forgotten, in my drafts folder for about a month.  Oops..

   -John

_______________________________________________
Int-area mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/int-area

---

From: "Bernard Aboba" <[email-masked]>
Date: Sat, 25 Aug 2007 22:23:44 -0700

Some additional comments:

Section 3.4

I am not comfortable with much of the discussion in this section.

As noted, much of the discussion is  academic because in fact, bit errors
do not happen in isolation.   Also, while it is possible to implement 
stronger frame
checking mechanisms, doing so at line rate (e.g. 10+ Gbps) is non-trivial.

If this material is intended to remain within the document, it needs to be
reviewed by IEEE 802.3 folks such as Pat Thaler.

Section 4.1

With the advent of Energy Efficient Ethernet,  it will be possible for link 
rate
to be adjusted in mid-conversation.   Therefore the having the allowed MTU
vary with link speed will mean that the MTU could conceivably change in
the middle of a conversation between two hosts.

Section 4.3

Having a layer 2 switch advertise its maximum MTU will not necessary provide
a host with useful information, since depending on the operation of (R)STP,
the switch may or may not be in the path.  Also, with Energy Efficient
Ethernet, one or more links along the path may adjust their speed;  if
the MTU were to vary, this would cause changes to the path MTU that
the host would not be to predict, even knowing the link speed-MTU
relationship of its edge switch.

>From: John Heffner <[email-masked]>
>To: Iljitsch van Beijnum <[email-masked]>
>CC: [email-masked]
>Subject: Re: [Int-area] draft-van-beijnum-multi-mtu-00
>Date: Sun, 26 Aug 2007 00:44:28 -0400
>
>mixed-MTU subnets.  I'm not yet convinced this draft constitutes a full and 
>practical solution to the problem.
>
>I have some specific comments below divided by section.  A few general ones 
>first:
>
>One thing that might make the draft easier to read would be some discussion 
>of how the pieces described in Section 4 fit together to form a solution to 
>the problem presented.
>
>The discussion in Section 3 is pretty good, but it might be a bit long and 
>detailed for this doc.
>
>
>4.1:
>
>It's not clear what exactly the Allowed MTU and Off-link MTU come from, and 
>how a router selects these values.  Is one of these the router's interface 
>MRU?  Why do you need an off-link MTU?  As long as you don't send packets 
>longer than the router's MRU, then PMTUD will take care of this.
>
>want links with a much slower speed to use smaller MTUs, but it seems like 
>the end hosts are able to make an appropriate decision.
>
>
>4.3:
>
>These are sent out every 5 minutes.  If a new device comes up, it might be 
>operating for 5 minutes (or MAYbe 60 seconds) without this information?  
>And, it's not even required for a host to listen to it. I'm not sure I 
>understand the justification for doing this at all.
>
>
>4.4:
>
>doesn't really say what to do with it.  Do you need to get a reply before 
>using a larger MTU?  If you don't get a reply at a given size, then what do 
>you do?  Fall back to the minimum?  Probe at other sizes a la 4821?
>
>seconds.  I don't see this as being practical (say, on a router interface) 
>where you may need to send/forward packets to large numbers of hosts that 
>may all need to be probed individually.
>
>
>Ultimately I would like to see something done in this area.  I might even 
>volunteer to help. :)
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 26 Aug 2007 17:06:53 +0200

On 26-aug-2007, at 6:44, John Heffner wrote:

> First, I'd like to say that I agree with the goal of being able to  
> run mixed-MTU subnets.

Excellent!

> I'm not yet convinced this draft constitutes a full and practical  
> solution to the problem.

Let's work on it then.

Based on earlier comments, I'm looking to simplify the next version  
of this draft.

> discussion of how the pieces described in Section 4 fit together to  
> form a solution to the problem presented.

Right.

> specifying 1500 byte minimum MTUs.  A standard in this area should  
> apply equally to all link types.

It's meant as a general mechanism. However, in reality, ethernet is  
pretty much the only broadcast network in town so that's why I  
mention it a lot. I'll see if I can focus more on the generality of  
the mechanisms.

> The discussion in Section 3 is pretty good, but it might be a bit  
> long and detailed for this doc.

I agree that this isn't appropriate in a protocol document. However,  
these issues have been brought up a lot in the past and I think many  
of the earlier arguments were flawed. I wanted to address those flaws  
as part of the discussion. I'll see if I can spin the CRC details off  
to a separate document at some point.

> It's not clear what exactly the Allowed MTU and Off-link MTU come  
> from, and how a router selects these values.  Is one of these the  
> router's interface MRU?  Why do you need an off-link MTU?  As long  
> as you don't send packets longer than the router's MRU, then PMTUD  
> will take care of this.

 From earlier in the document:

   Allowed MTU:
       The maximum MTU allowed administratively.

   Off-link MTU:
       The maximum packet size that is appropriate for communicating  
with
       off-link correspondents.

The allowed MTU is the maximum MTU the administrator will allow. So  
for instance, when a switch only supports 3000-byte packets, the  
administrator can set the allowed MTU to 3000 to make sure the  
optimum packet size is used without unnecessary probing at larger sizes.

The idea behind the off-link MTU is that hosts can use this value to  
base their TCP MSS and packet size to off-link destinations on so  
that PMTUD and possible problems related to it can be avoided easily  
while it's still possible to use large packets on the local subnet.  
Do you think this is unnecessary?

> The whole link speed thing makes me a bit uncomfortable.  You  
> definitely want links with a much slower speed to use smaller MTUs,  
> but it seems like the end hosts are able to make an appropriate  
> decision.

So do you think this should be removed? If the hosts don't make the  
right decision on their own and MTU size at a certain speed is a  
concern (i.e., VoIP jitter), this means that it's either necessary to  
advertise a conservative MTU for all speeds, or touch all host  
configurations.

> Since this is broadcast, a single switch can pull down the MTU for  
> an entire subnet.  Is that a good thing?

Although this could be useful if widely implemented, I think you're  
right and it's probably best to remove this part. If switches really  
want to advertise something they can always send out an RA with a  
lifetime of 0 so the switch won't be used as a default router.

> underspecified.  This section describes the MTU detection message,  
> but doesn't really say what to do with it.  Do you need to get a  
> reply before using a larger MTU?  If you don't get a reply at a  
> given size, then what do you do?  Fall back to the minimum?  Probe  
> at other sizes a la 4821?

I'll add text.

> You forbid sending MTU detection messages more often than once per  
> 60 seconds.  I don't see this as being practical (say, on a router  
> interface) where you may need to send/forward packets to large  
> numbers of hosts that may all need to be probed individually.

It's 60 seconds or until you get a reply back. What I want is to  
avoid generating large numbers of oversized packets, which could  
possibly trigger undesired behavior on devices that can't handle the  
larger packets. Maybe make this something that can be set  
administratively and suggest a default of 60 seconds?

> Ultimately I would like to see something done in this area.  I  
> might even volunteer to help. :)

Any help will be much appreciated, text, code, ASCII art, test  
devices that support really large MTUs...  :-)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 26 Aug 2007 17:42:50 +0200

On 26-aug-2007, at 7:23, Bernard Aboba wrote:

> do not happen in isolation.

Yes, it would be good to look at this in more detail.

> Also, while it is possible to implement stronger frame

With larger payloads, it could be interesting to explore checksums  
that derive their strength from a greater length rather than a strong  
algorithm. But if a relatively modest packet size works better at a  
certain point in time for a certain type of hardware, that's fine,  
that's the whole point of automatically using the maximum possible size.

> reviewed by IEEE 802.3 folks such as Pat Thaler.

Yes, it would be helpful to put the whole jumboframe/CRC issue to bed  
for good.

> With the advent of Energy Efficient Ethernet,  it will be possible  
> for link rate

There doesn't seem to be much information around about energy  
efficient ethernet. Depending on how it's implemented, this could be  
rather problematic for applications that need a certain quality of  
service. It is of course easy to limit the size of outgoing packets  
depending on link speed, even if the link speed is quite variable,  
but it's not exactly easy to do the same for incoming packets.  
Receiving a burst of 9000 byte packets when operating at 10 Mbps  
isn't exactly great for applications like VoIP. For IPv6 it would be  
possible to advertise a new, smaller MTU in a neighbor advertisement,  
but with IPv4 that's not as easy.

> relationship of its edge switch.

The idea is to listen to all broadcasts from switches and then use  
the largest packet size supported by them all. But I'm thinking it's  
probably easier to depend on test packets to make sure that packet  
sizes are limited to what switches support so I'll remove the switch  
advertisements in the next version.

---

From: John Heffner <[email-masked]>
Date: Sun, 26 Aug 2007 12:21:12 -0400

Iljitsch van Beijnum wrote:
>> router's interface MRU?  Why do you need an off-link MTU?  As long as 
>> you don't send packets longer than the router's MRU, then PMTUD will 
>> take care of this.
> 
>       The maximum packet size that is appropriate for communicating with
>       off-link correspondents.
> 
> The allowed MTU is the maximum MTU the administrator will allow. So for 
> instance, when a switch only supports 3000-byte packets, the 
> administrator can set the allowed MTU to 3000 to make sure the optimum 
> packet size is used without unnecessary probing at larger sizes.
> 
> base their TCP MSS and packet size to off-link destinations on so that 
> PMTUD and possible problems related to it can be avoided easily while 
> it's still possible to use large packets on the local subnet. Do you 
> think this is unnecessary?

Yes, I believe PMTUD works fine for this.

>> but it seems like the end hosts are able to make an appropriate decision.
> 
> right decision on their own and MTU size at a certain speed is a concern 
> (i.e., VoIP jitter), this means that it's either necessary to advertise 
> a conservative MTU for all speeds, or touch all host configurations.

Yes, I think it's probably unnecessary.  Devices (on both hosts and 
switches) should be able to automatically limit their MTUs based on link 
rate.  It may be worth talking about this some, and giving guidance as 
to what appropriate MTUs bounds might be for a given speed.

> It's 60 seconds or until you get a reply back. What I want is to avoid 
> generating large numbers of oversized packets, which could possibly 
> trigger undesired behavior on devices that can't handle the larger 
> packets. Maybe make this something that can be set administratively and 
> suggest a default of 60 seconds?

I still don't think this will help.  My opinion is that this 
responsibility needs to be pushed down a level.  I think any given 
network type needs to have its own set of rules.  For example, I'd say 
that with Ethernet, anything larger than 1500 byes should be forbidden 
when in half-duplex mode.  Switches and hosts should have reasonable max 
MTU sizes for each link speed to prevent excessive jitter and queue 
occupancy by a single packet.  In the event you have some buggy 
equipment such as a host interface that is known to behave badly when 
receiving oversized frames, it can be protected by administratively 
configuring its switch port to 1500 bytes.  While this will impose some 
administrative burden, I believe such cases are rare.

   -John

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 26 Aug 2007 19:07:45 +0200

On 26-aug-2007, at 18:21, John Heffner wrote:

[...]

It looks like our approaches are slightly different. I like to  
include all the information that could be useful in the protocol; you  
seem to favor a more sparse approach.

My philosophy is "overdesign and underimplement": make the design as  
complete as possible and then implement only those parts that you  
really need. This works a lot better than the other way around.  
Partially for this reason, I'm going to keep all of the mechanisms  
(except the switch advertisement) around but make as much of it  
optional as possible. Also, it's easier to discuss the merits of a  
protocol mechanism when that mechanism is laid out in the draft. We  
can then see if there is more that can be removed later on.

---

From: "Bernard Aboba" <[email-masked]>
Date: Sun, 26 Aug 2007 10:23:30 -0700

>Yes, it would be helpful to put the whole jumboframe/CRC issue to bed for 
>good.

Here are the minutes from a discussion with IEEE 802.3 on the subject of 
Jumbo Frames in August 2005:
http://www.iab.org/liaisons/ieee/2005-08-ieee802-liaison-report.html

>There doesn't seem to be much information around about energy efficient 
>ethernet.

Here is a link to the EEE WG within IEEE 802.3:
http://www.ieee802.org/3/eee_study/public/index.html

>The idea is to listen to all broadcasts from switches and then use  the 
>largest packet size supported by them all. But I'm thinking it's probably 
>easier to depend on test packets to make sure that packet sizes are limited 
>to what switches support so I'll remove the switch  advertisements in the 
>next version.

Yes, it makes sense to remove the switch advertisements.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 26 Aug 2007 19:30:22 +0200

On 26-aug-2007, at 19:23, Bernard Aboba wrote:

>> Yes, it would be helpful to put the whole jumboframe/CRC issue to  
>> bed for good.

> Here are the minutes from a discussion with IEEE 802.3 on the  
> subject of Jumbo Frames in August 2005:

Did anything happen in the IETF as a result of this?

---

From: "Bernard Aboba" <[email-masked]>
Date: Mon, 27 Aug 2007 06:16:47 -0700

>Did anything happen in the IETF as a result of this?

As far as I know, work on a Jumbo Frame document did not go forward within 
IETF.  However, I assume that the offer of assistance from IEEE 802.3 still 
remains valid.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 28 Aug 2007 07:51:44 -0700

> devices that support really large MTUs...  :-)

There was at least one earlier proposal to add MTU information
to RFC2461 Neighbor Discovery messages for tracking per-neighbor
MTUs on the same link:

  http://ietfreport.isoc.org/idref/draft-templin-ndiscmtu/

Fred
[email-masked]

---

From: Tony Li <[email-masked]>
Date: Mon, 27 Aug 2007 10:56:29 -0700

On Aug 27, 2007, at 6:16 AM, Bernard Aboba wrote:

> As far as I know, work on a Jumbo Frame document did not go forward  
> within IETF.  However, I assume that the offer of assistance from  
> IEEE 802.3 still remains valid.

To the best of my recollection, the work on Jumbo Frames culminated  
in http://tools.ietf.org/html/draft-ietf-isis-ext-eth-01 (and  
possibly one other draft that's now lost in the mists of time) and  
was primarily driven by one particular operator.  Given personnel  
turnover at the operator, I think that you'll find it hard to find a  
motivated constituency, especially given the chilly reception the  
first time.

Tony

---

From: Matt Mathis <[email-masked]>
Date: Wed, 29 Aug 2007 18:32:15 -0400 (EDT)

Tony,
draft-ietf-isis-ext-eth-01 predates the August 2005 report by several years.
This thing comes back every few years, and each time we get a little further.
(We being the pro-jumbo camp).

I have most recently asked for something that should be straight forward:
would the 802.3 vendors who claim to support jumbo please define what that
means, in a way that supports interoperability?  This sounds trivial, but is
not, because there are a large number ways of implementing Ethernet that have
implicit MTU limits.  And not too surprisingly, there are also know examples
of gear that only sort of delivers jumbo packets, or that does not
interoperate with other vendors.

In a simplistic view, such a document needs to say that the "clock rate
adaptation" is implemented by doing frame level store and forward, not a bit
or byte level FIFO.  I.e. The receiver needs to be able to collect an entire
frame at the maximum supported MTU even under the permitted worst case
tolerance for the sender and receiver clock rates.  It should also require a
settable, enforced MTU limit, so an network operator can choose the balance
between jitter, buffer resources and MTU.

All of the reputable vendors' products do this already.  But there are some
who don't.

Thanks,
--MM--
-------------------------------------------
Matt Mathis      http://www.psc.edu/~mathis
Work:412.268.3319    Home/Cell:412.654.7529
-------------------------------------------
Evil is defined by mortals who think they know
"The Truth" and use force to apply it to others.