Subject: [Int-area] Comments on draft-van-beijnum-multi-mtu-04

From: Pat Thaler <[email-masked]>
Date: Fri, 14 Nov 2014 19:56:16 +0000

Comments from an Ethernet point of view. These are just my personal comments. IEEE 802 should be asked to comment on this document, particularly 802.1 and 802.3 because of the references to Ethernet.

Abstract and 1 mention sending an Ethernet maximum frame size of 1500 bytes  and the referenced RFC 4321 in 7.2 mentions backing that off to 1400 bytes to allow for tunnels.  1500 bytes seems like a reference to the Ethernet maximum frame size (1518 with the MAC Header and CRC allows 1500 octets of payload).  In this context, the Frame format extension to IEEE Std 802.3 originally completed in the amendment IEEE Std 802.3as-2006  should probably be mentioned.

This added an additional maximum frame size of 1982 octets (2000 with the MAC header and CRC) to Ethernet to support "envelope frames" - frames  encapsulated with additional headers and trailers for purposes such as tunneling, security, etc.  This allows for encapsulation protocols adding up to 482 octets without requiring fragmentation (assuming that the original senders limit themselves to 1500 octet frames as intended.

Support for the larger frame size is common so generally 1500 byte packets plus a reasonable amount of encapsulation should go through.  Wince RFC 4321 was completed around the same time as IEEE 802.3as, it doesn't take the envelope maximum frame size into account, but now 8 years later there generally isn't a need to back off transmitted packet size from 1500 bytes to allow for tunnels.

The draft might mention this.

3. Defining Confirmed MTU as the largest packet successfully received or the largest packet acked assumes that the path is symmetric which it might not be. Equating MRU to MTU makes the same assumption. While often the L2 receive path is the same as the transmit path, that is not always the case.  This assumption is repeated elsewhere in the draft (e.g.  end of 9.2)

6. In cases where multipathing based on the 5-tuple is used, the test packet might not take the same path as the real traffic. For layer 2 bridging, VLAN tag can also affect path and it isn't mentioned in the link layer fields. Destination link-layer address is also omitted.

9.1 There are diminishing returns in efficiency as MTU size increases and the disadvantage such as head of line blocking and congestion increase. For example, Taking into account Ethernet overheads (IPG, preamble, MAC header), IPv6 header and TCP header, throughput is 87.7% for 1500 byte MTU, 97.8% for 9000 bytes and 98.9% for 18000 bytes.

9.3 A monitoring interval of 25 to 35 seconds on a high speed link implies that one is concerned about optimizing efficiency but willing to lose 30 seconds of data when the network changes so that the current MTU size isn't acceptable. That seems to be a contradiction..

11. Sending of  large packets based on a neighbor's willingness to accept them may degrade QoS for other traffic including traffic between other nodes on the subnet.  This could be a kind of denial of service to traffic that depends on low latency. This should be discussed as a security consideration.

Annex B doesn't mention that there can be physical layer limitations for large packets. Many  Ethernet PHYs have a clock skew FIFO to deal with differences in the transmitter's clock rate and the receiver's clock rate. Idles are dropped or inserted between packets to compensate for clock skew. The FIFOs may be sized for the standard maximum frame size. Because of the increment of idle insertion and deletion at speeds of 10 Gb/s and above, I expect that devices can cope with 9 Kbyte packets. But raising it to 64 Kbyte is likely to cause problems some of the time. This will not cause a packet loss every time because of variability of the IPG so test packets may get through. The likely outcome is packet loss much higher than the BER would produce but not high enough to trigger MTU adjustment.

B.1 The is no support provided for the assumption in the last paragraph. There are cases where higher speed links are used to reduce latency and latencies in the low microseconds are desired.

B.2  The statement:

"A crucial difference between PMTUD problems that result from MTUs

smaller than the de facto standard 1500 bytes and PMTUD problems that

result from MTUs larger than 1500 bytes is that in the latter case,

only the party that's actually using the non-standard MTU is

affected."

Is clearly untrue because traffic in higher classes of service experiences greater latency due to the over-size packets.

The assumption that MTU limitations will be near the edges is also unsubstantiated.

B.4, This section is incorrect. The Ethernet CRC has a hamming distance of 4 for up to the maximum packet size.  IEEE 802.3's design goal for undetected packet error rate requires a hamming distance of 4 (i.e. an 3 bit errors in a packet are detected).

According to "32-Bit Cyclic Redundancy Codes for Internet Applications" Philip Koopman, the Ethernet CRC has a hamming distance of 4 to 91067 bits (11,450 bytes).

There are no plans to make a negotiable FCS for Ethernet so that is a moot point. Some Ethernet PHYs have additional error protection to maintain the objectives of BER and undetected error rate with more complex transmission codes and on media with a higher uncorrected bit error rate, but that isn't being done with an objective of extending it to larger packet sizes.

B.5 There is no substantiation for the statements here. The disadvantage to small packet users may not be slight. Transmission rate ramping more quickly with large packet size may cause considerable congestion for all and take significant time to resolve. For those using large packets, TCP congestion control may not be able to reduce the rate enough. Have simulations or tests been conducted?

B.7 Conclusion - with my IEEE 802 hat off,  I agree that there are environments such as some data centers where 9000 byte jumbo frames will function fine, but I don't think they should be unleashed on the big I internet and I don't think that 64K jumbos are wise anywhere. Much of this draft doesn't seem consistent with this conclusion rather it seems to support much larger jumbo frames.  Mention of 9000 bytes and an applicability statement should be placed in the body of the draft, not left to the end of an annex.

Pat

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 16 Nov 2014 09:59:20 -0800

Pat,

Thanks for your extensive comments. This is very helpful.

On 14 Nov 2014, at 9:56, Pat Thaler <[email-masked]> wrote:

> Abstract and 1 mention sending an Ethernet maximum frame size of 1500 bytes  and the amendment IEEE Std 802.3as-2006  should probably be mentioned.

Yes, I'll mention it in the next version of the draft.

> 6. In cases where multipathing based on the 5-tuple is used, the test packet might not takeaffect path and it isn’t mentioned in the link layer fields. Destination link-layer address is also omitted.

In the presence of equal cost multipath (ECMP), i.e., some form of link aggregation, and load balancing is done on more than the layer 2 and IP addresses, it would be a big problem if the different paths support different maximum frame sizes. I can't think of any way to detect this to avoid problems.

I'm thinking that the best way to address this would be to add a mechanism that allows a system administrator to limit the MTU size for an entire subnet. (Some of your later comments also make me think this would be important.)

An obvious way to do this would be to use the RA MTU option, however, that one SHOULD be ignored if it contains a value larger than 1500. Adding a second MTU option that basically does the same thing would be ugly, but is perhaps the best way to address this.

> 9.3 A monitoring interval of 25 to 35 seconds on a high speed link implies that one is be a contradiction..

There is a tradeoff between reacting quickly and limiting the number of test packets. `maybe 30 seconds to repair a sudden decrease in MTU is too long. My thinking was that this would be exceedingly rare and only happen if the network experiences a disruption anyway, so a time that is short enough to keep TCP sessions alive would be sufficient.

> 11. Sending of  large packets based on a neighbor’s willingness to accept them may degrade QoS for other traffic including traffic between other nodes on the subnet.  This could be a kind of denial of service to traffic that depends on low latency. This should be discussed as a security consideration.

This is only an issue when traffic flows come together and start flowing over the same port. For switched Ethernet, that would/should not be the case for most traffic (assuming the backplane has sufficient capacity so that's not a bottleneck). The most common situation where that happens would be the port towards a router. If desired, the router could advertise a smaller MTU towards neighbors on the subnet.

But yes, this should be discussed somewhere.

> Annex B doesn’t mention that there can be physical layer limitations for large packets. Manytest packets may get through. The likely outcome is packet loss much higher than the BER would produce but not high enough to trigger MTU adjustment.

I'll see if I can find information about this. Obviously we don't want to increase the packet size so much that the error rate starts to increase.

> B.1 The is no support provided for the assumption in the last paragraph. There are cases microseconds are desired.

Ok.

> “A crucial difference between PMTUD problems that result from MTUs
> only the party that’s actually using the non-standard MTU is
> affected.” 

> Is clearly untrue because traffic in higher classes of service experiences greater latency due to the over-size packets.

But that's unrelated to path MTU discovery.

Also, it only happens when packet share a constrained path.

> B.4, This section is incorrect. The Ethernet CRC has a hamming distance of 4

I think you're right and I confused the number of flipped bits that will always be detected with the Hamming distance. I'll doublecheck and correct.

> B.5 There is no substantiation for the statements here. The disadvantage to small packet users may not be slight. Transmission rate ramping more quickly with large packet size may cause considerabl…

I'll look at the literature.

> B.7 Conclusion – with my IEEE 802 hat off,  I agree that there are environments such as some data centers where 9000 byte jumbo frames will function fine, but I don’t think they should be unleashed on the big I internet and I don’t think that 64K jumbos are wise anywhere. Much of this draft doesn’t seem consistent with this conclusion rather it seems to support much larger jumbo frames.  Mention of 9000 bytes and an applicability statement should be placed in the body of the draft, not left to the end of an annex.

Actually many backbones already support MTU sizes of 9000 or similar. I agree that there are parts of the internet where such large packets aren't appropriate, but I believe that correlates to a high degree with the link speed used.

Please note that I'm not advocating for 64k packets. However, if consenting adults decide that 64k is an appropriate packet size, I think our protocols should support that. Note that RFC2675 (IPv6 Jumbograms), which allows for a theoretical packet size of 4 GB, was published 15 years ago.

After reading your message, as mentioned before, I do come away with the notion that simply using whatever the hardware is capable of could be problematic in some situations, so we need something to limit the maximum packet for a given subnet.

As for placement in an annex: I'm now thinking the draft needs to be split into at least two documents, possibly more. The first one would be a problem statement, the second one my proposed protocol. I'll seek guidance from the chairs about that.

Iljitsch

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 17 Nov 2014 11:32:01 +1300

On 17/11/2014 06:59, Iljitsch van Beijnum wrote:
...
>> 6. In cases where multipathing based on the 5-tuple is
>> used, the test packet might not takeaffect path and it
>> isn’t mentioned in the link layer fields. Destination
>> link-layer address is also omitted.
> 
> In the presence of equal cost multipath (ECMP), i.e., some
> form of link aggregation, and load balancing is done on more
> than the layer 2 and IP addresses, it would be a big problem
> if the different paths support different maximum frame sizes.
> I can't think of any way to detect this to avoid problems.

Isn't this a pretty clear case of "don't do that!"? If an
operator configures ECMP (or LAG, unlikely) with different
MTUs on the paths, then PMTUD is almost certainly doomed.

...
> Please note that I'm not advocating for 64k packets. However,
> if consenting adults decide that 64k is an appropriate packet
> size, I think our protocols should support that. Note that
> RFC2675 (IPv6 Jumbograms), which allows for a theoretical
> packet size of 4 GB, was published 15 years ago.

With special technologies such as HIPPI in mind, and with
a specific statement:

"Jumbograms are relevant only to IPv6 nodes that may be attached
to links with a link MTU greater than 65,575 octets, and need
not be implemented or understood by IPv6 nodes that do not
support attachment to links with such large MTUs."

Clearly there was no expectation that PMTUD or fragmentation
would be relevant (remembering that "nodes" in IPv6 includes
hosts and routers).

A link MTU up to 65575 (65535+40) is *not* a jumbogram
in IPv6 terms.

   Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 16:49:49 +0000

Hi Brian an Iljitsch,

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Brian E Carpenter
> Sent: Sunday, November 16, 2014 2:32 PM
> To: Iljitsch van Beijnum
> Cc: [email-masked]
> Subject: Re: [Int-area] Comments on draft-van-beijnum-multi-mtu-04
> 

This one stood out prominently for me, too, and goes back to what I have
been saying about end-to-end vs hop-by-hop probing. Probes sent by the
end system itself (i.e., per RFC4821) have the same 5-tuple as ordinary
data packets and therefore will follow the same path. So, end-to-end
probing works across ECMP/LAG whereas hop-by-hop probing has no
idea of what path end-to-end data packets will take.

> would be relevant (remembering that "nodes" in IPv6 includes

Brian, I'm not sure why you have mentioned both here and several times
in the past that IPv6 Jumbograms should be out of scope (or maybe I am
mis-interpreting your point). RFC2675 exists, and if the mechanisms can
work equally well for jumbograms why not mention it in the document
instead of saying "out of scope"?

> A link MTU up to 65575 (65535+40) is *not* a jumbogram

Not in IPv6 terms, no, but in GigE terms 9KB is a "jumbogram".

Thanks - Fred
[email-masked]

>    Brian
> 
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 18 Nov 2014 08:34:16 +1300

Comment near the end...

> instead of saying "out of scope"?

Because RFC2675 is explicit that it only works for nodes that
are known to be connected by links that are known to support the
large link MTU. There is no suggestion that fragmentation or
PMTUD would be used, and certainly the people who campaigned for
IPv6 jumbograms expected their use to be handcrafted. Given that
the purpose is for maximising throughput in HPCN applications
(Big Science and the like) the last thing you want is fragmentation.

> Not in IPv6 terms, no, but in GigE terms 9KB is a "jumbogram".

Unfortunately that's true, so we have to be rather explicit in
what we write.

    Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 19:46:48 +0000

Hi Brian,

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Monday, November 17, 2014 11:34 AM
> To: Templin, Fred L
> Cc: Iljitsch van Beijnum; [email-masked]

Right, I would never suggest fragmenting an IPv6 Jumbogram (maybe we
should stress capitalizing the "J" to differentiate from GigE "jumbograms").
The protocol doesn't support it anyways.

I do believe that IPv6 PMTUD could be made to work over a path that includes
jumbo links (e.g., the path "4M -> 2M -> 1M -> 2M -> 4M" should result in a
Packet Too Big with MTU set to 1M). But, since no one is beating a path to my
door demanding jumbo support I will continue to not force the issue.

> Unfortunately that's true, so we have to be rather explicit in

Captial "J" for IPv6 Jumbos and little "j" or GigE "jumbos"?

Thanks - Fred

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 17 Nov 2014 20:57:53 +0100

On 17 Nov 2014, at 20:46, Templin, Fred L <[email-masked]> wrote:

> jumbo links

The ICMPv6 too big message has a 32-bit MTU field, so yes, it should work.

> Captial "J" for IPv6 Jumbos and little "j" or GigE "jumbos"?

That would be a terrible idea. And fortunately unnecessary, as it's jumboframes (Ethernet packets/frames with a payload larger than 1500 bytes) versus jumbograms (IPv6 packets with a payload larger than 65535 bytes).

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Nov 2014 20:23:52 +0000

Hi Iljitsch,

> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Monday, November 17, 2014 11:58 AM
> To: Templin, Fred L
> Cc: [email-masked]
> That would be a terrible idea. And fortunately unnecessary, as it's jumboframes (Ethernet packets/frames with a payload larger than
> 1500 bytes) versus jumbograms (IPv6 packets with a payload larger than 65535 bytes).

That would be fine, but I constantly see people using the shortened version "jumbo"
and you are still left wondering whether they mean jumbo as in IPv6 or jumbo as in
GigE. I will agree, however, that it is not easy to enforce capitalization nor to require
the long form of the word - so I guess we will just have to continue to rely on
evaluating the term according to the context in which it is used?

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Nov 2014 22:10:23 +0000

Hi Brian,

> MTUs on the paths, then PMTUD is almost certainly doomed.

Yes, it seems clear. But, is there a BCP anywhere that warns against non-uniform
ECMP? As in, what if one ECMP path has a 9KB MTU and another has 1280?

Thanks - Fred