Subject: [Int-area] Multi-MTU subnet draft

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 24 Feb 2008 16:25:43 +0100

Hi,

Here is an update of my multi-MTU subnet draft. Based on the  
discussion in the summer I've pruned the CRC text (although most of  
the points are still there) and I've spent a lot of time coming up  
with a way to make all of this simpler. I think I found a good tradeoff.

The basic idea is that routers advertise three parameters in IPv6  
router advertisements. (These parameters apply equally to IPv4 and  
IPv6 operation, though.) They are:

- An absolute maximum packet size that is allowed on a link. This can be
   used make sure larger packets aren't used if they're known to have  
bad
   effects (MAXMTU)

- A maximum packet size for nodes operating at 10 or 100 Mbps (the  
cutoff
   is 600 Mbps) in case these are connected to switches that don't  
support
   jumboframes or it's undesirable that slow nodes send large packets  
for
   QoS reasons (SLOWMTU)

- A packet size cutoff for not probing / probing for a jumbo-clean path
   (SAFEMTU)

The neighbor discovery options and jumbo ARP for communicating per- 
neighbor MTUs and probing whether large packets work are still there,  
but they are optional. Hosts may use RFC 4821 as a probing mechanism  
instead. (Although this means that some transport protocols can use  
larger packets and others can't and routers can't use RFC 4821.)

Alternatively, nodes may forego probing altogether if they stick to a  
maximum packet size of SAFEMTU. The disadvantage of this is that  
SAFEMTU must be set administratively because only the administrator  
can know what a safe MTU is in the absense of probing. But the good  
part is that it is compatible with current manually configured jumbo  
frame use.

There needs to be a bit more text about operational issues (or maybe  
those should go in an applicability document), and I'm not sure yet if  
I want to keep the CRC discussion even though it's a lot shorter now.  
Depending on the discussion on the list and in Philadelphia I want to  
see if this can be published as experimental.

Begin forwarded message:

> A New Internet-Draft is available from the on-line Internet-Drafts  
> directories.

> 	Title           : Extensions for Multi-MTU Subnets
> 	Author(s)       : I. van Beijnum
> 	Filename        : draft-van-beijnum-multi-mtu-02.txt
> 	Pages           : 17
> 	Date            : 2008-02-24

> In the early days of the internet, many different link types with
> many different maximum packet sizes were in use.  For point-to-point
> or point-to-multipoint links, there are still some other link types
> (PPP, ATM, Packet over SONET), but shared subnets are now almost
> exclusively implemented as ethernets.  Even though the relevant
> standards mandate a 1500 byte maximum packet size for ethernet, more
> and more ethernet equipment is capable of handling packets bigger
> than 1500 bytes.  However, since this capability isn't standardized,
> it's seldom used today, despite the potential performance benefits of
> using larger packets.  This document specifies a mechanism for
> advertising a non-standard maximum packet size on a subnet.  It also
> specifies optional mechanisms to negotiate per-neighbor maximum
> packet sizes so that nodes on a shared subnet may use the maximum
> mutually supported packet size between them without being limited by
> nodes with smaller maximum sizes on the same subnet.

> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-02.txt

_______________________________________________
Int-area mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 28 Feb 2008 12:08:39 -0800

Iljitsch,

Apologies on being slow to coming around to reading this.
Still have to read further, but one question for now: is
this useful for router-to-router, or is it only used for
host-to-router (and maybe also host-to-host). For that
matter, perhaps the definition of what is meant by "host"
and "router" is somewhat soft?

Thanks - Fred
[email-masked]  

>-----Original Message-----
>From: Iljitsch van Beijnum [mailto:[email-masked]] 
>Sent: Sunday, February 24, 2008 7:26 AM
>To: Internet Area
>Subject: [Int-area] Multi-MTU subnet draft
>
>Hi,
>
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 28 Feb 2008 23:23:28 +0100

> and "router" is somewhat soft?

It's meant for each of the following cases:

router-router
router-host
host-host

I've tried to be careful with using the words "node" and "host". Most  
stuff applies both to routers and hosts, but there are a few things  
that only hosts can do because they get to determine the size of  
packets that are sent, routers have no choice in the matter. I.e., the  
TCP MSS option makes sure a jumbo-capable host doesn't send TCP  
segments larger than the standard MTU to a non-jumbo-capable host, and  
RFC 4821 also only works on hosts.

So should implementers want to keep their life simple and omit the  
neighbor discovery options and jumbo ARP, they can take care of the  
difference through RFC 4821. But routers can't send larger packets  
unless this is administratively configured OR the neighbor discovery /  
jumbo ARP probing takes place.
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Feb 2008 14:39:11 -0800

> RFC 4821 also only works on hosts.

It would be very useful to indicate when you're talking about a device 
or the behavior of a device, i.e.:
	host vs 'device acting as a host'
It might be more useful to just call that "endpoint" and to call the 
other thing a "forwarder", AND to note that whether a node is an 
endpoint or a forwarder depends on its behavior to a particular packet, 
NOT a property of the device itself.

The reason is that routers often act as endpoints (for control plane, 
e.g.); if we say "routers never do X", then a router acting as an 
endpoint will never do it, which would be bad. A recent case is 
transport mode IPsec, which ought to be required for control plane 
support, but is considered optional because, well, it's a "security 
gateway", not a "host".

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 29 Feb 2008 00:12:09 +0100

On 28 feb 2008, at 23:39, Joe Touch wrote:

> It would be very useful to indicate when you're talking about a  
> device or the behavior of a device, i.e.:

"a device acting as a" is implied.

> The reason is that routers often act as endpoints (for control  
> plane, e.g.); if we say "routers never do X", then a router acting  
> as an endpoint will never do it, which would be bad.

I'll see if further clarification is required in this area in the next  
version. However, I think I was fairly careful with router, node and  
host. Most of the time the distinction is just that routers send  
router advertisements and hosts listen for them.
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Feb 2008 15:22:37 -0800

> "a device acting as a" is implied.

Sure - although I appreciate you're being careful here, it's important 
that the output clarify this issue. We're living with the cost of not 
doing so in other docs...

>> endpoint will never do it, which would be bad.
> 
> host. Most of the time the distinction is just that routers send router 
> advertisements and hosts listen for them.

Is there, e.g., never a case where you'd want a router to act like a 
host while it's being configured?

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 28 Feb 2008 17:36:44 -0800

Iljitsch,

Having read further now, I think I get what is being
proposed. As I mentioned before, I have also suggested
adding MTU info in IPv6 ND messages in past efforts:

 http://www.potaroo.net/ietf/idref/draft-templin-v6v4-ndisc/
 http://www.potaroo.net/ietf/idref/draft-templin-ndiscmtu/

If I get what is being proposed, I believe the mechanisms
you are proposing may be useful in conjunction with SEAL.

The reason I asked about the router-to-router is that
that is the SEAL use case, and it would be "interesting"
to see whether the loose interpretation of RS/RA you are
making could be applied between a pair of routers at
either end of a virtual link such as for the SEAL
ingress/egress tunnel routers.

A couple of specific comments about your doc:

1) Section 1, text beginning: "However, [RFC4821] must
be implemented in every transport protocol...", I'm not
sure I understand/agree with the point. My understanding
of RFC4821 was that incremental deployment was supported.
Also, I don't understand the part about "significant
probability for failures"?

2) Section 4.1 and onwards, I'm not so sure about the
"SLOWMTU" applicability. IMHO, the mechanisms in this
document should be useful whether the receiver of the
RA is directly connected to the same physical link as
the router that sent it, or whether there are potentially
many physical links with different properties in the
path (e.g., when the subnet spans multiple bridged links.)

3) Section 4.2, perhaps this text is part of what you were
meaning about being careful about node, host, route, etc.
Particularly interesting is the part beginning: "Devices
not acting as IPv6 routers ... MAY send out a router
advertisement ...". I think this is worth considering,
and also worth considering is whether two routers connected
by a virtual link that spans an interdomain routing region
(default-free, perhaps) can mutually send RAs with multi-MTU
options to one another (with Router Lifetime=0, of course).
The Ras might also contain RFC4191 RIOs, but no RFC4861 PIOs
and such.

What do you think?

Thanks - Fred

>Sent: Thursday, February 28, 2008 2:23 PM
>To: Templin, Fred L
>Cc: Internet Area
>Subject: Re: [Int-area] Multi-MTU subnet draft
>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 29 Feb 2008 16:36:14 +0100

On 29 feb 2008, at 0:22, Joe Touch wrote:

>> However, I think I was fairly careful with router, node and host.  
>> Most of the time the distinction is just that routers send router  

> host while it's being configured?

Well, it would be good if you could set up a maximum MTU for a subnet  
in a single place. One way to do that is to have one router advertise  
it and the others listening for the advertisements. But I'm not sure  
that is a good solution. Alternatively, the routers could learn this  
information from a server such as a DHCP server. But that is even more  
dangerous: what if the server becomes unreachable?

Ideally, the routers should implement the full protocol including  
probing so they don't have to configured with a static maximum,  
they'll figure it out dynamically.
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 29 Feb 2008 17:05:15 +0100

> http://www.potaroo.net/ietf/idref/draft-templin-ndiscmtu/

I saw one of them before, yes. But the approach is fairly different.

> ingress/egress tunnel routers.

Right. That should work well if neighbor discovery is performed  
through the tunnel, which is fairly common from what I've seen.

One issue would be that if the tunnel gets rerouted, the MTU can  
change. The draft currently only catches that with dead neighbor  
detection, which routers on tunnel endpoints can't do... So that could  
be a problem. In theory the same problem can happen on a layer 2  
network but in practice that wouldn't happen very easily.

> of RFC4821 was that incremental deployment was supported.

Well, if you have RFC 4821 in your TCP implementation that doesn't buy  
you anything for NFS or DNS with EDNS0 over UDP. It's incremental in  
the sense that you can deploy it where it's not deployed and things  
get better. But in the case of my draft, we make things worse by  
removing the fixed MTU so if you then repair that with something that  
doesn't cover 100% of all cases, you're left with a certain number of  
cases where there will be problems.

> probability for failures"?

"if hosts implementing [RFC4821] interact with hosts that don't  
implement this mechanism but do use a larger than standard MTU."

Suppose host A lives on a subnet with variable MTUs and host B lives  
on a different subnet with a manually configured jumboframe MTU. If A  
and A's router support probing as outlined in the draft, then  
everything will work.

However, suppose there is a switch on the network that doesn't support  
A's MTU. Without the probing defined in the draft, A can still  
discover what packet size it can use towards B, but the router doesn't  
see that A can't successfully receive the large packets that B sends  
to it can't send back a too big, and B doesn't implement RFC 4821 so  
if it sends large packets, it can't recover from the path MTU  
discovery black hole.

> path (e.g., when the subnet spans multiple bridged links.)

Absolutely. But what does that have to do with SLOWMTU?

The idea behind SLOWMTU is twofold: it could be that the slow hosts  
are behind switches that don't support the larger MTU, and even if the  
slow hosts COULD send large packets, that may be undesirable because  
it increases RTTs and jitter.

> options to one another (with Router Lifetime=0, of course).

Why would they send out router advertisements? The information about  
the supported MTUs is available in neighbor solicitations/ 
advertisements, which routers DO send to each other.

> The Ras might also contain RFC4191 RIOs, but no RFC4861 PIOs

Hey, a new neighbor discovery RFC... Managed to miss that.

RIO = route information option, PIO = prefix information option (or  
something like that)? Note that these acronyms don't appear in those  
RFCs.

I think it could be useful for routers to advertise a maximum packet  
size for the default route, or at least, indicate that large packets  
can't be used there. The downside is that you can then only use them  
on the local subnet, unless you start adding them to RFC 4191  
information... Sticking to PMTUD is simpler. Packet sizes aren't  
dependent on addresses so in the prefix options wouldn't make much  
sense, IMO.
_______________________________________________
Int-area mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 29 Feb 2008 08:52:33 -0800

Iljitsch,

>Sent: Friday, February 29, 2008 8:05 AM
>To: Templin, Fred L
>Cc: Internet Area

There are many others I could cite as well. But, the approach
is to put extra "stuff" in ND messages that pertains to MTU.

>network but in practice that wouldn't happen very easily.

I'm not looking for the RA to tell an MTU size that will
work across the tunnel once and for always. I want the RA
coming from the tunnel far end to tell the tunnel near end
the MRU the TFE is able to receive w/o respect to the tunnel
path MTU. That way, the TNE can tell whether it even makes
sense to admit a large packet into the tunnel since there
is no point in doing so if the TFE's MRU is too small.

>you anything for NFS or DNS with EDNS0 over UDP.

Oh - I thought when you said "every transport protocol"
you were talkng about every transport protocol on all
nodes everywhere. It is clear now that you were talking
about every transport protocol *of the same node*. I'm
not sure on that point though, because I thought that
at one point RFC4821 had something to say about MTU
sharing between transports of the same node. Need to
check to see if it is still there...

>It's incremental in  

>if it sends large packets, it can't recover from the path MTU  

This isn't a failure case for running or not RFC4821, though;
this is a failure case for allowing nodes with diverse MTUs
on links that don't support diverse MTUs. Indeed, this failure
case has nothing to do with RFC4821.

>it increases RTTs and jitter.

I am considering the case of fast links at the edges and
slow links in the middle. Something like GigE at the edges
and 100Mbps Ethernet in the middle, i.e., the "dumbell
configuration" yet again! If the nodes on fast links think
they can send large packets based on their attached link
bandwidths, it could be a problem.

>advertisements, which routers DO send to each other.

I thought you were saying the NS/NA functions were optional?
Also routers sending RAs to each other can be used as a means
to propagate RFC4191 Route Information Options (RIOs).

>Hey, a new neighbor discovery RFC... Managed to miss that.

No; not at all. Standard ND combined with RFC4191.

Fred