Subject: Path MTU Hop-by-Hop Option

From: Bob Hinden <[email-masked]>
Date: Tue, 16 Oct 2018 15:55:22 -0700

Hi,

Gorry and I have submitted a new proposed hop-by-hop option that records the minimum Path MTU from the source to a destination host.  The summary in the abstract is:

  This document specifies a new Hop-by-Hop IPv6 option that is used to
  record the minimum Path MTU from a source to a destination host.
  This collects a minimum recorded MTU along the path to the
  destination.  The value can then be communicated back to the source
  host by an ICMPv6 Packet Too Big message.

  This Hop-by-Hop option is intended to be used in environments like
  Data Centers and on paths between Data Centers, to allow them to
  better take advantage of paths able to support a large Path MTU.

This came out of some discussions about how to improve path MTU discovery.    The idea is that it can provide additional feedback to transport mechanisms.  We don’t think this is a panacea to the Internet's Path MTU problems, but think it might be helpful.   We see more immediate use in datacenters where large MTU links exist and the operators of these datacenters have more control over the hosts and routers in the datacenters.

 Please read the draft.    Links below.

Thanks,
Bob

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-hinden-6man-mtu-option-00.txt
> Date: October 16, 2018 at 3:40:06 PM PDT
> To: "Godred Fairhurst" <[email-masked]>, "Robert Hinden" <[email-masked]>, "Robert M. Hinden" <[email-masked]>, "Gorry Fairhurst" <[email-masked]>
> 
> 
> A new version of I-D, draft-hinden-6man-mtu-option-00.txt
> has been successfully submitted by Robert M. Hinden and posted to the
> IETF repository.
> 
> Name:		draft-hinden-6man-mtu-option
> Revision:	00
> Title:		IPv6 Minimum Path MTU Hop-by-Hop Option
> Document date:	2018-10-16
> Group:		Individual Submission
> Pages:		10
> URL:            https://www.ietf.org/internet-drafts/draft-hinden-6man-mtu-option-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-hinden-6man-mtu-option/
> Htmlized:       https://tools.ietf.org/html/draft-hinden-6man-mtu-option-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-hinden-6man-mtu-option
> 
> 
> Abstract:
>   This document specifies a new Hop-by-Hop IPv6 option that is used to
>   record the minimum Path MTU from a source to a destination host.
>   This collects a minimum recorded MTU along the path to the
>   destination.  The value can then be communicated back to the source
>   host by an ICMPv6 Packet Too Big message.
> 
>   This Hop-by-Hop option is intended to be used in environments like
>   Data Centers and on paths between Data Centers, to allow them to
>   better take advantage of paths able to support a large Path MTU.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 16 Oct 2018 16:58:54 -0700

On Tue, Oct 16, 2018 at 3:55 PM, Bob Hinden <[email-masked]> wrote:
> Hi,
>
> Gorry and I have submitted a new proposed hop-by-hop option that records the
> minimum Path MTU from the source to a destination host.  The summary in the
> abstract is:
>
> This came out of some discussions about how to improve path MTU discovery.
> The idea is that it can provide additional feedback to transport mechanisms.
> We don’t think this is a panacea to the Internet's Path MTU problems, but
> think it might be helpful.   We see more immediate use in datacenters where
> large MTU links exist and the operators of these datacenters have more
> corol over the hosts and routers in the datacenters.
>

Hi Bob,

I like the concept, but have the concern it relies on both extension
headers and ICMP to get through the network. Alternatively would it
make sense for the peer host to reflect the value received in another
option? For instance, TCP syn is set with option to probe MTU,
receiver sends syn-ack with the option that states what the received
MTU value is.

Tom

> To: "Godred Fairhurst" <[email-masked]>, "Robert Hinden"
> <[email-masked]>, "Robert M. Hinden" <[email-masked]>, "Gorry
> Fairhurst" <[email-masked]>
>
>
> URL:
> https://www.ietf.org/internet-drafts/draft-hinden-6man-mtu-option-00.txt
> Status:
> https://datatracker.ietf.org/doc/draft-hinden-6man-mtu-option/
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-hinden-6man-mtu-option
>
>
> Abstract:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Fred Baker <[email-masked]>
Date: Tue, 16 Oct 2018 17:36:41 -0700

draft-hinden-6man-mtu-option?

You need two options, or something that acts like two options. In the forward direction, in the HBH header, the option collects the path attribute. In the reverse direction, it reports the path attribute. That could be the same option in the destination options header if you prefer. If you said that, I looked, and I missed it.

I would suggest talking with Dina and/or Nandita about XCP and RCP. What they were looking for was a way to estimate the optimal throughput rate, which divided by PMTU yields wind size. That of course is not "the bandwidth of the slowest link en route", but "the *available* bandwidth of the slowest link en route, mumble mumble season to taste".

> On Oct 16, 2018, at 3:55 PM, Bob Hinden <[email-masked]> wrote:
> 
> Signed PGP part
> Hi,
> 

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: "Lubashev, Igor" <[email-masked]>
Date: Wed, 17 Oct 2018 02:36:36 +0000

> From: Tom Herbert <[email-masked]>
> Sent: Tuesday, October 16, 2018 7:59 PM
> 
> I like the concept, but have the concern it relies on both extension headers
> and ICMP to get through the network. Alternatively would it make sense for
> the peer host to reflect the value received in another option? For instance,
> TCP syn is set with option to probe MTU, receiver sends syn-ack with the
> option that states what the received MTU value is.

I would recommend against sending any new extension headers during handshake (for any protocol).  A packet dropped during handshake will cause the connection establishment to be delayed.  Instead, such extension headers can be used after the connection has been established and with packets that will not cause a hiccup to the connection if dropped.  For TCP, it would be an ACK with no data for this particular proposal.

- Igor

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 17 Oct 2018 06:49:50 +0200

On Oct 17, 2018, at 00:55, Bob Hinden <[email-masked]> wrote:
>   record the minimum Path MTU

A while ago I proposed something similar to collect information about adaptation layer fragmentation MTUs (information about which is truly not available in any other way):

https://tools.ietf.org/html/draft-bormann-intarea-alfi-04

(I also only looked at the Internet layer here — leaving it as an exercise to put in transport/application protocol support.)

After mulling an initial version of the proposal, there was pretty much consensus that hop-by-hop options were not really available on the Internet.  So I put in an alternative way to collect the information, see above if you are interested.  Then at some point SPUD came along and I was hoping that would lead to a framework within in which this information could be better handled.

Maybe it’s time to pull the old draft out again.  The hop-by-hop option problem of course hasn’t gone away.

Grüße, Carsten

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 17 Oct 2018 08:44:57 +0200 (CEST)

> Please read the draft.  Links below.

I believe the router should not only report the MTU of the outgoing 
interface, it should also report MTU and MRU of the incoming interface 
(whatever is smallest, or both).

I have seen misconfigurations where each side of a link has different 
MTU/MRU, resulting in PMTUD blackhole with silently dropped packets. If 
we're serious about this option, we should report whatever is lowest of 
all MTU/MRU values of incoming and/or outgoing interfaces this packet 
touched. That would potentially involve including all interfaces this 
packet could have ECMP:ed egress interface.

It would also be interesting to have a mechanism along the lines of 
https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05 to actually 
have the routers probe the working MTU between each other, and report 
that. Whatever option you propose should take this kind of mechanism into 
account (and use the probed value if that's lower than the interface 
setting).

Also, what value should be reported in case of MPLS and varying other 
encapsulations. I guess the value of the IPv4/IPv6 packet that could fit 
into the egress interface after all necessary labels etc are put on there?

So if the egress encap path is two labels deep and the egress MTU is 9180, 
then 9172 should be reported as IPv4/IPv6 egress MTU as returned by the 
router when processing this proposed option?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 17 Oct 2018 08:31:09 +0100

On 17/10/2018, 00:58, Tom Herbert wrote:
> On Tue, Oct 16, 2018 at 3:55 PM, Bob Hinden<[email-masked]>  wrote:
>> Hi,
>>
Indeed, currently, feedback relies on some form of ICMP Message 
(currnetly  a PTB Message) from the remote endpoint. We could have 
instead reflected the received PMTU in a hop-by-hop option or a 
transport layer field.

Before, we consider other ways, I am curious *why* we think that an ICMP 
message from the destination endpoint would not reach the sender. This 
is a very different pathology from that of a router on-path generating 
an ICMP message, and I expect this often works, do other people have 
reasons why this wouldn't work, which should start me to be concerned?

Gorry
>> To: "Godred Fairhurst"<[email-masked]>, "Robert Hinden"
>> <[email-masked]>, "Robert M. Hinden"<[email-masked]>, "Gorry
>> Fairhurst"<[email-masked]>
>>
>>

---

From: Havard Eidnes <[email-masked]>
Date: Wed, 17 Oct 2018 10:12:24 +0200 (CEST)

> Before, we consider other ways, I am curious *why* we think
> that an ICMP message from the destination endpoint would not
> reach the sender. This is a very different pathology from that
> of a router on-path generating an ICMP message, and I expect
> this often works, do other people have reasons why this
> wouldn't work, which should start me to be concerned?

Both have the same root cause, and it's the same cause which has been
causing un-ending deployment issues for TCP path MTU discovery for
IPv4 and complicated and slow workarounds such as "TCP path MTU black
hole discovery".  An important reason is that people are paranoid
(e.g. because sometime in the 90s(?), one platform had a vulnerability
called "ping of death", and ICMP is therefore forever tarred as
security-problematic, never mind that the original problem is long
gone) and as a consequence they block ICMP on their border, ignoring
any and all protocol consequences.

Some people might call this "stupid", but the practice is sufficiently
widespread to cause significant problems.

Regards,

- Håvard

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 17 Oct 2018 09:34:52 +0100

> widespread to cause significant problems.
If I understand correctly, this isn't a "problem" that causes a
failure or side-effect, it simply means that devices that block ICMP
do not benefit from the method. That's not good, but if others
do benefit then that need not in itself mean we shouldn't do this.

Personally, I think the only hope is to confirm what works at the
transport layer, so to me this is only "extra" information that
can help save a lot of work when it is available.

Gorry

---

From: Jen Linkova <[email-masked]>
Date: Wed, 17 Oct 2018 19:35:19 +1100

> any and all protocol consequences.

I'm not sure it's the only/main root cause for PMTUD problems
currently. From my experience most router platforms have hardware
limits for generating ICMP error messages and those limits are much
lower than pps being served by those routers. So most of affected
senders never receive ICMP errors as those errors are never generated.
So generating ICMP at the endpoint would definitely help.

-- 
SY, Jen Linkova aka Furry

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 17 Oct 2018 10:37:56 +0200 (CEST)

On Wed, 17 Oct 2018, Gorry Fairhurst wrote:

> transport layer, so to me this is only "extra" information that can help 
> save a lot of work when it is available.

I fully agree with this. Considering all other things that can go wrong, 
this proposal is only an addition to methods that can be used. Considering 
the load on routers to handle hop-by-hop options I'm not even sure it's 
worth while to deploy it. I would rather OS vendors spent time 
implementing PMTU blackhole detection and mitigation than try to implement 
support for this hop-by-hop option, if I need to choose one of them.

---

From: Ole Troan <[email-masked]>
Date: Wed, 17 Oct 2018 10:43:34 +0200

> So generating ICMP at the endpoint would definitely help.

https://tools.ietf.org/html/rfc4443#section-2.4 bullet f
https://tools.ietf.org/html/rfc4443#section-5.2 bullet 4

Cheers,
Ole

---

From: Jen Linkova <[email-masked]>
Date: Wed, 17 Oct 2018 20:07:58 +1100

> https://tools.ietf.org/html/rfc4443#section-5.2 bullet 4

Sure. However I suspect that the difference between "pps need to be
generated to respond to all affected sources" and "pps allowed by the
rate limiting policy" would be much higher for infrastructure routers
than for destination systems.

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 17 Oct 2018 11:40:11 +0200

Bob Hinden wrote on 17/10/2018 00:55:
> Gorry and I have submitted a new proposed hop-by-hop option that records 
> the minimum Path MTU from the source to a destination host.  The summary 
> in the abstract is:

It would be useful to make it explicit that this only deals with the 
minimum path MTU in the forward path from one host to another.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 17 Oct 2018 11:50:04 +0200

> So generating ICMP at the endpoint would definitely help.

Is HBH processing going to be given a higher default bucket priority 
than generating icmp error messages?

In general, this ID is a difficult proposition, mostly because if you 
get a path MTU incorrect, it causes havoc.  Set against this, the data 
collection mechanism is acknowledged to be unreliable.  So there's an 
inherent mismatch in here between the mechanism and the expected 
reliability of the result.

Paths also change without warning, which means that unless the path mtu 
is constantly measured, data will end up being blackholed.

Nick

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Wed, 17 Oct 2018 12:14:32 +0000

Well put, Mikael. Both MTU and MRU are important for PMTU. And in case of incoming interface and outgoing interface of a router, those values can be different.  

And the assumption that two routers connected via a link will likely have the same IP MTU in a sane design is not always true. We have seen the non-compliance crop up time to time even in well managed networks. 

At minimum, it should be made clear what the host cares for and what the routed network reports - MTU vs IP MTU vs MPLS MTU. 

Having said all that, it is a bit puzzling to expect any network giving HBH option so much more tender and making this ICMP solution more efficient than the others. 

Cheers,
Rajiv  

>> On Oct 17, 2018, at 2:45 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> 
> I believe the router should not only report the MTU of the outgoing interface, it should also report MTU and MRU of the incoming interface (whatever is smallest, or both).
> 
> I have seen misconfigurations where each side of a link has different MTU/MRU, resulting in PMTUD blackhole with silently dropped packets. If we're serious about this option, we should report whatever is lowest of all MTU/MRU values of incoming and/or outgoing interfaces this packet touched. That would potentially involve including all interfaces this packet could have ECMP:ed egress interface.
> 
> It would also be interesting to have a mechanism along the lines of https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05 to actually have the routers probe the working MTU between each other, and report that. Whatever option you propose should take this kind of mechanism into account (and use the probed value if that's lower than the interface setting).
> 
> Also, what value should be reported in case of MPLS and varying other encapsulations. I guess the value of the IPv4/IPv6 packet that could fit into the egress interface after all necessary labels etc are put on there?
> 
> So if the egress encap path is two labels deep and the egress MTU is 9180, then 9172 should be reported as IPv4/IPv6 egress MTU as returned by the router when processing this proposed option?
> 
> --

---

From: Michael Richardson <[email-masked]>
Date: Wed, 17 Oct 2018 09:28:37 -0400

> I would recommend against sending any new extension headers during
    > handshake (for any protocol).  A packet dropped during handshake will
    > cause the connection establishment to be delayed.  Instead, such
    > extension headers can be used after the connection has been established
    > and with packets that will not cause a hiccup to the connection if
    > dropped.  For TCP, it would be an ACK with no data for this particular
    > proposal.

Right. You've described PLPMTU.  I use it daily.
Works great.  Just wish it was on by default everywhere.

My understanding is that it's not popular with high-end web systems that do
hardware offload, where it's simply cheaper in terms of Tx-opportunities to
just send more packets.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Mark Andrews <[email-masked]>
Date: Thu, 18 Oct 2018 01:34:15 +1100

> On 18 Oct 2018, at 12:28 am, Michael Richardson <[email-masked]> wrote:
> 
> 

It works well for TCP.  It does not work for UDP…

People need top fix their firewalls.  CPEs need to stop doing MSS fixups so that the errors are made more visible.  Stop papering over the misconfigurations.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Wed, 17 Oct 2018 14:41:37 +0000

Hi, this document should cite RFC1063 which was an early IP PMTUD approach
based on IP options with the minimum MTU recorded at each hop. It fell by the
wayside when RFC1191 was published.

Also, the definition of a new IPv6 hop-by-hop option for MTU discovery was
proposed in 2003/2004 in  'draft-welzl-pmtud-options':

https://datatracker.ietf.org/doc/draft-welzl-pmtud-options/
https://www.ietf.org/mail-archive/web/pmtud/current/msg00082.html

The approach operated in the same spirit as for this new document, but
as I recall it did not get traction in the PMTUD working group and also fell
by the wayside.

Maybe there is something new in this new document (e.g., the sending
of the ICMP PTB) that would give it a chance of succeeding where the
early approaches did not?

Fred

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 17 Oct 2018 11:03:32 -0400

What happens if there is a tunnel in the middle?
If it is just IP in IP, I suppose you could require copying the 
information back and forth between the headers.

Does this work at all if there are is MPLS in the path?  What about 
other tunnels?

Yours,
Joel

On 10/16/18 6:55 PM, Bob Hinden wrote:
> This came out of some discussions about how to improve path MTU 
> discovery.    The idea is that it can provide additional feedback to 
> transport mechanisms.  We don’t think this is a panacea to the 
> Internet's Path MTU problems, but think it might be helpful.   We see 
> more immediate use in datacenters where large MTU links exist and the 
> operators of these datacenters have more control over the hosts and 
> routers in the datacenters.
> 
>> *From: [email-masked] <mailto:[email-masked]>
>> *Subject: **New Version Notification for 
>> draft-hinden-6man-mtu-option-00.txt*
>> *Date: *October 16, 2018 at 3:40:06 PM PDT
>> *To: *"Godred Fairhurst" <[email-masked] 
>> <mailto:[email-masked]>>, "Robert Hinden" <[email-masked] 
>> <mailto:[email-masked]>>, "Robert M. Hinden" 
>> <[email-masked] <mailto:[email-masked]>>, "Gorry 
>> Fairhurst" <[email-masked] <mailto:[email-masked]>>
>>
>>
>> Name:draft-hinden-6man-mtu-option
>> Revision:00
>> Title:IPv6 Minimum Path MTU Hop-by-Hop Option
>> Document date:2018-10-16
>> Group:Individual Submission
>> Pages:10
>> URL: 
>> until the htmlized version and diff are available at tools.ietf.org 
>> <http://tools.ietf.org>.
>>
>> The IETF Secretariat
>>
> 
> 
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 17 Oct 2018 16:15:19 +0100

If the method worked, a single (or at least few) appropriately-size 
probe(s) could be sent to determine a PMTU.

If the path actually has a smaller effective MTU - e.g. due to encaps 
headers that were not known - it is just the same as a layer 2 device 
that wasn't correctly configured, or an IPv6 router that does not 
process the option. In other words, the hint provided by the method 
could actually be wrong. I'd suggest it is always up to the PL to 
discover the lack of connectivity at the PMTU size and choose a better 
size to avoid black holing data. This helps to slect one probe size.

Gorry

On 17/10/2018, 16:03, Joel M. Halpern wrote:
>> Gorry and I have submitted a new proposed hop-by-hop option that 
>> records the minimum Path MTU from the source to a destination host. 
>>  The summary in the abstract is:
>>
>>> <mailto:[email-masked]>>, "Robert Hinden" 
>>> <[email-masked] <mailto:[email-masked]>>, "Robert M. 
>>> Hinden" <[email-masked] <mailto:[email-masked]>>, "Gorry

---

From: Tom Herbert <[email-masked]>
Date: Wed, 17 Oct 2018 08:16:36 -0700

On Wed, Oct 17, 2018, 12:31 AM Gorry Fairhurst <[email-masked]> wrote:

> records the
> discovery.

Gorry,

ICMP filtering is still common at firewalls. If HBH succeeds over a path in
one direction, there's a good chance it works in the reverse path so
bidirectional probability of success should be close to P. ICMP has it's
own success rate say PI. Using both HBH and ICMP then has success rate of
something like P*PI. P*PI < P most likely.

Besides, reflecting the information in an option is a simpler solution. No
extra packets are sent, less risk of DOS attack, and don't need to rate
limit. It does assume bidirectional communication.

Tom

> https://tools.ietf.org/html/draft-hinden-6man-mtu-option-00
> >> Htmlized:

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 17 Oct 2018 16:20:13 +0100

On 17/10/2018, 16:16, Tom Herbert wrote:
>
>
> On Wed, Oct 17, 2018, 12:31 AM Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     > On Tue, Oct 16, 2018 at 3:55 PM, Bob Hinden<[email-masked]
>     <mailto:[email-masked]>>  wrote:
>     >> Hi,
>     >>
> ICMP filtering is still common at firewalls. If HBH succeeds over a 
> path in one direction, there's a good chance it works in the reverse 
> path so bidirectional probability of success should be close to P. 
> ICMP has it's own success rate say PI. Using both HBH and ICMP then 
> has success rate of something like P*PI. P*PI < P most likely.
>
> Besides, reflecting the information in an option is a simpler 
> solution. No extra packets are sent, less risk of DOS attack, and 
> don't need to rate limit. It does assume bidirectional communication.
>
> Tom
>
Thanks, I see this logic for the feedback. I'll look for measurements to 
back-up this intuition.

If you know of any measurements, let me know.

Gorry
>     >> From: [email-masked] <mailto:[email-masked]>
>     >> To: "Godred Fairhurst"<[email-masked]
>     >> <[email-masked] <mailto:[email-masked]>>, "Robert
>     M. Hinden"<[email-masked] <mailto:[email-masked]>>, "Gorry
>     >> Fairhurst"<[email-masked] <mailto:[email-masked]>>
>     >>
>     >>
>     >> until the htmlized version and diff are available at
>     tools.ietf.org <http://tools.ietf.org>.
>     >>
>     >> The IETF Secretariat
>     >>
>     >>
>     >>
>     >>
>     >> [email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 17 Oct 2018 08:33:43 -0700

On Tue, Oct 16, 2018, 9:50 PM Carsten Bormann <[email-masked]> wrote:

> A while ago I proposed something similar to collect information about
> adaptation layer fragmentation MTUs (information about which is truly not
> available in any other way):
>
> (I also only looked at the Internet layer here — leaving it as an exercise
> to put in transport/application protocol support.)
>
> After mulling an initial version of the proposal, there was pretty much
> consensus that hop-by-hop options were not really available on the
> Internet.  So I put in an alternative way to collect the information, see
> above if you are interested.  Then at some point SPUD came along and I was
> hoping that would lead to a framework within in which this information
> could be better handled.
>

Carsten,

SPUD would put this sort of information in a UDP payload. This would then
have intermediate nodes modifying payloads to record the MTU. That's
fundamentally incorrect since port numbers, which would be used to identify
SPUD, only have meaning at the end points.

If intermediate nodes are to set arbitrary information in packets, then
Hop-by-Hop options are really the only correct and generic way to do that.

> problem of course hasn’t gone away.

As the draft states, RFC8200 relaxed the requirements of HBH options so
that intermediate nodes ignore them instead of dropping. That is reasonable
and easy to implement. There are still going to be some drops so
implementations will need to fall back when they don't work for a flow
(same thing would be true SPUD since UDP isn't guaranteed to work across
the whole Internet).

Tom

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 17 Oct 2018 17:44:23 +0200 (CEST)

On Thu, 18 Oct 2018, Mark Andrews wrote:

> People need top fix their firewalls.  CPEs need to stop doing MSS fixups 
> so that the errors are made more visible.  Stop papering over the 
> misconfigurations.

This is not only done in home gateways, this is done in lots of other 
places.

And no, this can't be fixed by degrading the service for a substantial 
amount of people who have fully working service today (as far as they can 
tell). You can't willfully degrade customers service and expect them to 
complain about it and then through customer support get fault reports and 
then fix it. No sane management person will accept that. It's hugely 
costly both in opex and customer satisfaction.

This is also an network engineering knowledge issue in that lots of people 
just don't have the knowledge to test these things and understand how it's 
supposed to work. I've had the discussion so many times regarding PMTUD 
not working and people just saying "oh, we'll just use TCP MSS adjust to 
fix that" and when I complain that this breaks non-TCP protocols I 
basically get "yeah, so what?" back.

Getting PMTUD working to 100% is never going to work. It's too fragile and 
have too many failure causes. The ones people typically bring up here is 
just a fraction of all the differnet ones I have run into over the past 20 
years.

---

From: George Michaelson <[email-masked]>
Date: Wed, 17 Oct 2018 17:47:52 +0200

QUIC does UDP-like interior tasting of the viable path to get a
packetsize. Or did: It may have moved to a pragmatically low value
(its very UDP-like as it exposes on the wire, even if it has
session/transport behaviours)

I think there is a level at which people throw up their hands and say
jumbo is never going to happen but I remain an optimist, and dream
that we could have a respectful Layer-2 conversation (we tolerate
MPLS) about appropriate L2 transfer units, and why we need to move
them up stream from 1500 even inside the home, and to the CPE.

Given the "death of transit" story, I think a case can be made that
for a significant amount of content, given tight binding from that
content to the ISP, the path is both short, and in control of at most
two agencies: the one which owns the CAN, and the one which owns the
link into the CDN. If they just sat and talked, 2k and 4k end to end
would be perfectly possible.

dreaming of a larger packet future...

-G

---

From: Bob Hinden <[email-masked]>
Date: Wed, 17 Oct 2018 15:52:34 -0700

George,

> On Oct 17, 2018, at 8:47 AM, George Michaelson <[email-masked]> wrote:
> 

That was, essentially, part of our thinking that this might be useful in environments where there is more control over the infrastructure.  From the abstract:

and described in more detail in Section 3.   If it can be proven useful there, then perhaps it could be useful on the rest of the Internet.  Getting 9K packets to work in and between data centers itself is a reasonable win in our opinion.

> 
> dreaming of a larger packet future…

Haven’t given up :-)

Bob

---

From: Bob Hinden <[email-masked]>
Date: Wed, 17 Oct 2018 15:54:28 -0700

> On Oct 17, 2018, at 2:40 AM, Nick Hilliard <[email-masked]> wrote:
> 
> It would be useful to make it explicit that this only deals with the minimum path MTU in the forward path from one host to another.

This is described pretty well in the Introduction.  There is even a figure :-)

Bob

---

From: Bob Hinden <[email-masked]>
Date: Wed, 17 Oct 2018 16:01:53 -0700

Tom,

> On Oct 17, 2018, at 8:20 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
>> ICMP filtering is still common at firewalls. If HBH succeeds over a path in one direction, there's a good chance it works in the reverse path so bidirectional probability of success should be close to P. ICMP has it's own success rate say PI. Using both HBH and ICMP then has success rate of something like P*PI. P*PI < P most likely.
>> 
>> Besides, reflecting the information in an option is a simpler solution. No extra packets are sent, less risk of DOS attack, and don't need to rate limit. It does assume bidirectional communication.
>> 
>> Tom
>> 
> Thanks, I see this logic for the feedback. I'll look for measurements to back-up this intuition.
> 

I wanted to add that we are not tied to using ICMP Packet Too Big to convey the path mtu information back to the source.  If there is a mechanism that has a higher probability of getting to the source, it would only help.

The basic idea is to use the HBH option to get the path MTU information to the destination, and then have it  send it back to the source.   We think this may work better than having the routers in the middle send this feedback.

Bob

---

From: Tom Herbert <[email-masked]>
Date: Wed, 17 Oct 2018 17:49:24 -0700

On Wed, Oct 17, 2018, 4:01 PM Bob Hinden <[email-masked]> wrote:

> path in one direction, there's a good chance it works in the reverse path
> so bidirectional probability of success should be close to P. ICMP has it's
> >> Besides, reflecting the information in an option is a simpler solution.
> No extra packets are sent, less risk of DOS attack, and don't need to rate
> I wanted to add that we are not tied to using ICMP Packet Too Big to
> convey the path mtu information back to the source.  If there is a
> mechanism that has a higher probability of getting to the source, it would
> only help.
>
> The basic idea is to use the HBH option to get the path MTU information to
> the destination, and then have it  send it back to the source.   We think
> this may work better than having the routers in the middle send this
> feedback.
>

Agreed, reflecting this sort of information from the end host is the only
way to go. ECN already works in that mode, and I imagine there will be
other use cases like oam that will do that. Option reflection is also
essential to FAST. Using the same mechanism in both directions seems
preferred, and if routers are producing or consuming the information then
HBH is only way to go.

Tom

---

From: Christian Huitema <[email-masked]>
Date: Thu, 18 Oct 2018 10:11:40 +0200

> session/transport behaviours)

Several implementations of QUIC send "PMTU probes" -- basically QUIC
level PING with variable length padding, and with the DF bit set. The
packets do not contain any application level data, so have very little
side effect apart from congestion control. There is no specific
algorithm on how and when these packets should be sent, but the protocol
supports it, and the mechanism is fairly simple: if a packet carrying N
bytes is acknowledged, the implementation infers that the PMTU is at
least N.

It seems robust but of course nothing is perfect. It can be too
optimistic if the network is load balancing between two paths with
different PMTU. And then there is the issue of what happens if the PMTU
changes over time.

-- Christian Huitema

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 18 Oct 2018 11:19:37 +0200

Bob Hinden wrote on 18/10/2018 00:54:
>> On Oct 17, 2018, at 2:40 AM, Nick Hilliard <[email-masked]>
>> wrote: Bob Hinden wrote on 17/10/2018 00:55:
>>> records the minimum Path MTU from the source to a destination
>>> host.  The summary in the abstract is:
>> 
>> It would be useful to make it explicit that this only deals with
>> the minimum path MTU in the forward path from one host to another.
> 

yes, but not named explicitly :-)

It might be clearer if the first sentence read:

    This draft proposes a new Hop-by-Hop Option to be used to record the
    minimum MTU along the forward path between the source and destination
    nodes.

Nick

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 18 Oct 2018 11:41:08 +0100

>    minimum MTU along the forward path between the source and destination

Or perhaps even more explicit?

- saying it ... records the configured Link MTU in IPv6 nodes that 
implement the method ...?

Gorry

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 14:36:18 +0200 (CEST)

> dreaming of a larger packet future...

It's completely undeployable in real life without something like 
https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05

It's not realistic to expect a whole lan of managed and unmanaged devices 
to show up at a LAN and communicate with each other on a large MTU LAN.

Right now my only tools are to crank up the L2 switch to 9000 MTU and 
advertise MTU=9000 in RA. Ok, I show up with my device that only has 1500 
MTU/MRU, I see that RA and I'm like "well, I can't do higher than 1500 
anyway, so I'll continue doing that". I need to send a UDP packet to the 
other guy on the same LAN, that's fine, he has 9000 MRU/MTU. He then needs 
to send me 2000 bytes back, so he does that and because my MRU is only 
1500, my NIC now silently drops that packet. No PMTUD, just silent drop. 
*poof* gone.

There are LOTS of similar failure modes that can happen. We need 
per-destination PMTU both on/off-link. I need to be able to send RIO and 
say "ok, ::/0 has 1492 PMTU" while at the same time send PIO with 
"MTU 9000". Devices then need to understand if their neighbours actually 
can do 9000 or not.

I've even seen hardware/software that will have interface MTU of 1500, 
take the RA MTU=9000, set it's IPv6 interface MTU to 9000, set MSS to 
89xx, send a large packet (because it can), then it borks when a response 
comes back because it only has ~2000 in MRU.

Yes, real life is that broken.

---

From: George Michaelson <[email-masked]>
Date: Thu, 18 Oct 2018 14:46:08 +0200

dreams become nightmares very quickly.

I will observe that moving to jumbo for a locally attached file-store
was painful, so I do know even in the purely local context I've hit
pain.

However, in constructing GPON common-bearer services (NZ, AU) the
issue of a layer-2 which can encompass 1500 end to end, with a mix of
different overlay provider mechanisms (PPPoE, PPPoATM, Native,
VLAN...) It also becomes clear, the engineers who run the fibre are
well aware they have technology in the wide, talking from CPE to ISP
router, which *could* do it. They have to. They are pretty much
obliged to handle 2k, to allow the interior to be encap/decap for
1500. Customrs still get 14xx but the actual back-end is now north of
that. I think it could be more north.

Its not the glass and the layer-2 won't do "big" any more, its how the
upper layers are configured in equipment which is the problem.

What worries me is the people who are going to chime in about chipsets
and SFP designs which *cannot* do it. I suspect there is far more
transmission equipment stuck in the past than I would like.

I'll shut up now. I just wish as an SDO we could segment this problem
off and get a rational conversation about pits and poles to fix this.

-G

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 15:20:58 +0200 (CEST)

> transmission equipment stuck in the past than I would like.

Actually, from that aspect it looks good. Most devices today can do at 
least ~2000. However, I still consider this "baby jumbo". There are 
non-trivial amount of devices out there that will not do 9000, especially 
in the residential gateway space. Wifi as far as I know won't even do more 
than ~2300.

> off and get a rational conversation about pits and poles to fix this.

I want to fix this, but to make the dream come true there are lots of 
things that need to be solved. Per-ND entry MTU and per-route MTU are two 
cruical things I have identified so far. The per-ND one is the most 
important that we're missing today, so we can have mixed-MTU/MRU hosts on 
the same LAN.

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 18 Oct 2018 15:32:57 +0200

> implement the method ...?

wfm.  It helps to provide explicit statements to the reader rather than 
leaving them to work out what is meant.

Nick

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 18 Oct 2018 10:22:29 -0400

> in the residential gateway space. Wifi as far as I know won't even do more

https://www.raspberrypi.org/forums/viewtopic.php?t=52807

not a residential gateway (well, you could but..) .. I imagine that there
are a LOT of things which are:
  1) not even 1G capable (think building management-like things, for
instance.. or power management things.. or a slew of the 'iot' things)
  2) even if they are 1G, their upper-layer parts may not (for a long
while) support jumbo

food for thought anyway.

> > off and get a rational conversation about pits and poles to fix this.
you mean the IEEE?

> important that we're missing today, so we can have mixed-MTU/MRU hosts on
sounds fun, sounds like madness in a LAN of more than a few devices though
(state management on all the connected entities seems challenging when the
device count gets large - maybe not at under 1k devices?)

---

From: Philip Homburg <[email-masked]>
Date: Thu, 18 Oct 2018 17:04:40 +0200

>device count gets large - maybe not at under 1k devices?)

Assuming that ND would provide this information (i.e. part of NS/NA is to
figure out the largest frame size that gets through), then why do you think
this would create additional complexity?

Thinking a bit out of the box, what if we would deprecate both
fragmentation and also any explicit support for PMTU detection (current ICMP
packet too big).

And then tell any higher level protocol to solve the issue at that level.

For example, DNS over UDP could just truncate at 1280. Optionally with sending
a bigger reply first.

TCP could for example start at 1280 and then try to probe for higher segment
sizes (as opposed to the current behavior where it starts with the
negotiatiated MSS and lowers as a result of PMTU detection).

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 15:09:38 +0000

Hi Mikael,

> -----Original Message-----
> From: ipv6 [mailto:[email-masked]] On Behalf Of Mikael Abrahamsson
> Sent: Thursday, October 18, 2018 6:21 AM
> To: George Michaelson <[email-masked]>
> Cc: [email-masked]
> Subject: Re: operational stoppers to > 1500 MTU deployment in real life (Re: Path MTU Hop-by-Hop Option)
> 

The problem with multi-MTU is link-scoped multicast. The interface must have
an MTU that is no smaller than the maximum per-ND entry MTU. Multicast
applications will see the interface MTU (not the per-ND entry MTU) and can
then send multicast packets that are too large for some neighbors.

Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 15:20:46 +0000

About larger MTUs in general, I know of at least one application whose performance
is highly sensitive to the size of the UDP datagrams it sends. So much so that the app
intentionally invokes IP fragmentation in order to send 64K UDP datagrams. Even with
IP fragmentation, the performance for sending a 64K UDP datagram is substantially
greater than sending 1500 UDP datagrams without fragmentation.

So, a brighter future would include links and paths with larger MTUs. But, to use
them, we need a more robust PMTUD capability. I don't know whether/not this
approach is the right one - others have tried it in the past but did not get traction.
But, we definitely need something (i.e., beyond just RFC[1191,1981,4821]).

Fred

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 18 Oct 2018 17:21:36 +0200

> this would create additional complexity?

operationally, mtu problems have always been a bag of pain.  If the 
underlying complexity associated with configuring them goes from O(1) to 
O(n^2) where n is the number of devices talking to each other, 
operational problems will also increase.  There is great value in 
simplicity, and any decision to move away from a simpler system needs to 
be taken in the context of risk and long term cost/benefit.

Nick

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 17:48:45 +0200

> On 18 Oct 2018, at 17:20, Templin (US), Fred L <[email-masked]> wrote:
> 
> greater than sending 1500 UDP datagrams without fragmentation

Hard to imagine any other reason than poor implementation for that. 

Cheers 
Ole

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 16:16:13 +0000

Ole,

> From: Ole Troan [mailto:[email-masked]]
> Sent: Thursday, October 18, 2018 8:49 AM
> To: Templin (US), Fred L <[email-masked]>
> Cc: [email-masked]

No, the implementation is fine. The reason is that (at least on linux) the "sendmsg()"
system call for SOCK_DGRAM can only send one UDP datagram from user space to
kernel space at a time. So, to increase performance, the application is obliged to
send the largest possible UDP datagram in a single system call.

We have been experimenting with "sendmmsg()" which allows the sending of multiple
UDP datagrams in a single system call, but so far have not seen better performance.
We need to understand this better.

Note that this is different than SOCK_STREAM (i.e., TCP).  With TCP, the application
simply stuffs data into the socket buffer and TCP segmentation happens within the
kernel (or within H/W assist). So, more data can be crammed into the socket buffer
in a single system call, and the application can keep the queue full while the kernel
consumes it.

Thanks - Fred

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 18:50:25 +0200 (CEST)

On Thu, 18 Oct 2018, Templin (US), Fred L wrote:

> The problem with multi-MTU is link-scoped multicast. The interface must 
> have an MTU that is no smaller than the maximum per-ND entry MTU. 
> Multicast applications will see the interface MTU (not the per-ND entry 
> MTU) and can then send multicast packets that are too large for some 
> neighbors.

That's fine. We're anyhow generally talking about mixing devices with MTU 
of 1500 and up, not down.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 18:51:59 +0200 (CEST)

> be taken in the context of risk and long term cost/benefit.

The proposal doesn't really involve any increase in configuration.

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 17:02:32 +0000

Mikael,

> From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: Thursday, October 18, 2018 9:50 AM
> Subject: RE: operational stoppers to > 1500 MTU deployment in real life (Re: Path MTU Hop-by-Hop Option)
> 

If you have three neighbors with MTUs 1500, 4K and 8K you are obliged to set
the interface MTU to 8K. The IP layer will then see the 8K interface MTU and
think it can send link-scoped multicasts that large. The interface can't return
a PTB because it is a link-scoped packet.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 18 Oct 2018 10:19:22 -0700

On Thu, Oct 18, 2018, 9:17 AM Templin (US), Fred L <
[email-masked]> wrote:

> UDP datagrams in a single system call, but

And the analog recvmmsg().

so far have not seen better performance.
> We need to understand this better.

Yes, we would expect sendmmsg to give better performance in that case.

> in a single system call, and the application can keep the queue full while

TCP can also use segmentation offload. This is a big win, and one of the
reasons TCP consistently outperforms UDP. For instance, as far as the stack
is concerned 64K packets can be sent or received. It's only very low in
driver or device that MTU sized segments are created (TX) or coalesced (RX).

No concept of a stream in UDP means we can't do this, however it's
conceivable it could be done if there's a ULP running over UDP. QUIC is a
candidate, however encryption of transport layer makes it challenging.
Crypto offload is possible benefit for that.

Tom

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 19:24:41 +0200

Still sounds like an implementation detail and not a “UDP requires IP fragmentation support to achieve performance “. 

Cheers 
Ole

> On 18 Oct 2018, at 19:19, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
>> On Thu, Oct 18, 2018, 9:17 AM Templin (US), Fred L <[email-masked]> wrote:
>> Ole,
>> 
> TCP can also use segmentation offload. This is a big win, and one of the reasons TCP consistently outperforms UDP. For instance, as far as the stack is concerned 64K packets can be sent or received. It's only very low in driver or device that MTU sized segments are created (TX) or coalesced (RX).
> 
> No concept of a stream in UDP means we can't do this, however it's conceivable it could be done if there's a ULP running over UDP. QUIC is a candidate, however encryption of transport layer makes it challenging. Crypto offload is possible benefit for that.
> 
> Tom
> 
>> 
>> Thanks - Fred
>> 
>> > Cheers
>> > Ole
>>

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 17:31:52 +0000

Tom,

Thanks for repeating (and thereby confirming) what I said.

Fred

From: Tom Herbert [mailto:[email-masked]]
Sent: Thursday, October 18, 2018 10:19 AM
To: Templin (US), Fred L <[email-masked]>
Cc: Ole Troan <[email-masked]>; 6man <[email-masked]>
Subject: Re: operational stoppers to > 1500 MTU deployment in real life (Re: Path MTU Hop-by-Hop Option)

On Thu, Oct 18, 2018, 9:17 AM Templin (US), Fred L <[email-masked]<mailto:[email-masked]>> wrote:
Ole,

> From: Ole Troan [mailto:[email-masked]<mailto:[email-masked]>]
> To: Templin (US), Fred L <[email-masked]<mailto:[email-masked]>>
> Cc: [email-masked]<mailto:[email-masked]>
> > On 18 Oct 2018, at 17:20, Templin (US), Fred L <[email-masked]<mailto:[email-masked]>> wrote:
> >

Tom

Thanks - Fred

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]<mailto:[email-masked]>
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 19:47:09 +0200 (CEST)

On Thu, 18 Oct 2018, Templin (US), Fred L wrote:

> If you have three neighbors with MTUs 1500, 4K and 8K you are obliged to 
> set the interface MTU to 8K. The IP layer will then see the 8K interface 
> MTU and think it can send link-scoped multicasts that large. The 
> interface can't return a PTB because it is a link-scoped packet.

We'd of course have to come up with a solution that doesn't cause this 
problem.

And no, I don't agree that's what you have to do. You can keep the default 
MTU at 1500 and raise the MTU for certain neighbours, or you set the 
interface MTU to 8000 and set the default IPv4/IPV6 MTU to 1500. These are 
typically all different values on several platforms.

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 17:47:23 +0000

I didn’t say “UDP requires IP fragmentation”. I said that larger UDP datagrams give better
performance regardless of whether/not IP fragmentation is used. I am sure you would
agree that it would be best if we could send large UDP datagrams w/o needing any IP
fragmentation. So, larger MTUs would benefit performance.

Thanks - Fred

From: Ole Troan [mailto:[email-masked]]
Sent: Thursday, October 18, 2018 10:25 AM
To: Tom Herbert <[email-masked]>
Cc: Templin (US), Fred L <[email-masked]>; 6man <[email-masked]>
Subject: Re: operational stoppers to > 1500 MTU deployment in real life (Re: Path MTU Hop-by-Hop Option)

Cheers
Ole

On 18 Oct 2018, at 19:19, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:

Tom

Thanks - Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 17:54:37 +0000

Mikael,

IP interfaces expose only a single MTU to the IP layer. In my example,
if the neighbors had MTUs of 1500, 4K and 8K then the interface MTU
needs be set to 8K. For non-link-local packets, the interface can then
return locally-generated PTBs if the IP layer tries to send an 8K packet
to either the 1500 or 4K neighbors. But, for link-scoped packets, it
would break the model of a "link" if the interface were to return
a PTB. The link model is described in RFC3819.

Fred

> Sent: Thursday, October 18, 2018 10:47 AM

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 20:28:07 +0200 (CEST)

> IP interfaces expose only a single MTU to the IP layer. In my example,

Yes and no.

This is valid configuration and can be achieved in both Linux kernel and 
on routers from several large vendors.

Interface blah
mtu 9180
ipv6 mtu 9000
ipv4 mtu 1500

In the Linux kernel, interface MTU and the MTU exposed to each protocol 
can be different (each IP protocol MTU can be set per-interface using 
/proc settings). I can also use the "ip route" command to add an 
interface route to a certain IP and set a different MTU to that specific 
IP address, regardless if it's on-link or off link (via a router).

> a PTB. The link model is described in RFC3819.

I'm fine if the interface MTU is 9180, IPv6 default MTU is whatever is 
seen in the RA standard MTU option (or if we define an extension to it), 
and then when I detect a neighbor that supports the ND MTU discovery 
mechanism, I can now talk larger packets with that specific next-hop after 
probing that it actually works. Did you read the draft I linked to the 
other day?

RFC3819 is overly pessimistic in its views on how things can work.

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 20:35:19 +0200

> On 18 Oct 2018, at 20:28, Mikael Abrahamsson <[email-masked]> wrote:
> 
> This is valid configuration and can be achieved in both Linux kernel and on routers from several large vendors.
> 
> Interface blah
> mtu 9180
> ipv6 mtu 9000
> ipv4 mtu 1500
> 
> In the Linux kernel, interface MTU and the MTU exposed to each protocol can be different (each IP protocol MTU can be set per-interface using /proc settings). I can also use the "ip route" command to add an interface route to a certain IP and set a different MTU to that specific IP address, regardless if it's on-link or off link (via a router).
> 
> I'm fine if the interface MTU is 9180, IPv6 default MTU is whatever is seen in the RA standard MTU option (or if we define an extension to it), and then when I detect a neighbor that supports the ND MTU discovery mechanism, I can now talk larger packets with that specific next-hop after probing that it actually works. Did you read the draft I linked to the other day?
> 

Indeed. In VPP, MTU is a per adjacency property. 
Per-neighbour/per-protocol MTU, should be doable. Anyone up for implementing it at BK hackathon? (The Iljitsch draft)

Cheers 
Ole

---

From: 神明達哉 <[email-masked]>
Date: Thu, 18 Oct 2018 11:42:10 -0700

At Thu, 18 Oct 2018 19:24:41 +0200,
Ole Troan <[email-masked]> wrote:

> Still sounds like an implementation detail and not a “UDP requires
> IP fragmentation support to achieve performance “.

This may not be about "performance", and in any case I'm not sure if
it's relevant to this thread, but DNS servers generally rely on IPv6
fragmentation because
- A DNS server can send pretty large (e.g. >1500B) responses
- It usually responds to an arbitrarily large number of unpredictable
  client addresses over a non-connected UDP socket
- It may depending on the OS implementation, but at least some OS do
  not use ICMPv6 PMTU too big errors unless there's a corresponding
  transport layer connection.  So PMTUD wouldn't help even if the
  ICMPv6 message was delivered.
- Even if PMTUD works for that OS implementation, it still involves a
  timeout and retry at the client side; the DNS server implementation
  can't expect to get notified about the error for the unconnected UDP
  socket, and even if it can get it, it has already discarded the
  original response due to the stateless nature of the protocol.  The
  resulting delay is often considered unacceptable.

So many existing DNS server implementations have the OS fragment large
responses at the size of IPv6 minimum MTU.  Even if the OS doesn't
support the API to use the minimum MTU, those implementations still
just hope the MTU of the outgoing interface is the PMTU and
fragmentation at that size is enough, rather than trying to, e.g.,
trigger a fall back to TCP.

This might still be considered an "implementation detail", but I think
it's also an issue common for any implementation due to the protocol
characteristic of DNS.

--
JINMEI, Tatuya

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 18 Oct 2018 20:52:45 +0200 (CEST)

> Indeed. In VPP, MTU is a per adjacency property.

PMTUD needs a per-destination cached table of probed MTUs where these are 
less than the default next-hop MTU. This is how it's done in the Linux 
kernel. It used to be done by adding a route with the probed MTU. So this 
adjacency MTU can be done exactly like that, from user-space. From what I 
know, no Linux kernel modifcations are needed. Would perhaps be nicer if 
this was in the kernel, but it's not strictly needed.

> Per-neighbour/per-protocol MTU, should be doable. Anyone up for implementing it at BK hackathon? (The Iljitsch draft)

I forwarded your email to Iljitsch. When I last talked to him a year ago, 
he indicated he already had started an implementation. He might be able to 
provide more details about it.

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 20:59:25 +0200

Jinmei.

> it's also an issue common for any implementation due to the protocol

I think this is tangential. Here you have an application where PMTUD doesn’t work,
and it doesn’t implement segmentation/reassembly at the application level.
Some would consider this broken.
See https://datatracker.ietf.org/doc/draft-ietf-intarea-frag-fragile/

Ole

---

From: Tom Herbert <[email-masked]>
Date: Thu, 18 Oct 2018 12:00:41 -0700

On Thu, Oct 18, 2018 at 10:24 AM, Ole Troan <[email-masked]> wrote:
> Still sounds like an implementation detail and not a “UDP requires IP
> fragmentation support to achieve performance “.
>
I also tend to think that techniques segmentation offload like this
mostly eliminate the need for jumbo frames. That's one of those ideas
that seem to popup in the big DC operators periodically. The
conclusion, after much discussion, always seem to be the same: in the
end the benefits don't outweigh the costs.

Tom

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 19:32:45 +0000

Hi Mikael,

> Sent: Thursday, October 18, 2018 11:28 AM

That is still only one MTU for the IP layer, even if there are different values
for IPv6 and IPv4. Talking multi-access interfaces, the IP layer sees a single
value for the interface.

> IP address, regardless if it's on-link or off link (via a router).

Sure, and on multi-access interfaces the MTU for each destination would
be less than or equal to the MTU of the interface.

> probing that it actually works.

But, you can't send larger than the interface MTU.

>Did you read the draft I linked to the other day?

Didn't see it, but happy to take a look.

> RFC3819 is overly pessimistic in its views on how things can work.

RFC3819 is the BCP. So, you would either have to update or obsolete it.

Thanks - Fred

---

From: "Dale W. Carder" <[email-masked]>
Date: Thu, 18 Oct 2018 15:35:31 -0400

> end the benefits don't outweigh the costs.

Perhaps, for consumer-grade use cases.  Today there are in fact sizable 
deployments of 9k endpoints used for data transfers on a global scale
where for these high-BDP flows, MSS matters.  To keep this working there
are test probes doing active end-to-end measurements along the paths.  I
think this draft or something like it is a nice step in the right
direction. 

Dale

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 19:35:45 +0000

Ole,

> Sent: Thursday, October 18, 2018 11:35 AM
> To: Mikael Abrahamsson <[email-masked]>
> Cc: Templin (US), Fred L <[email-masked]>; [email-masked]
> > In the Linux kernel, interface MTU and the MTU exposed to each protocol can be different (each IP protocol MTU can be set per-
> interface using /proc settings). I can also use the "ip route" command to add an interface route to a certain IP and set a different MTU
> to that specific IP address, regardless if it's on-link or off link (via a router).
> >
> > I'm fine if the interface MTU is 9180, IPv6 default MTU is whatever is seen in the RA standard MTU option (or if we define an
> extension to it), and then when I detect a neighbor that supports the ND MTU discovery mechanism, I can now talk larger packets
> with that specific next-hop after probing that it actually works. Did you read the draft I linked to the other day?
> >

I don't know VPP, but it sounds like multiple point-to-point interfaces instead
of a single multi-access interface. Certainly, each point-to-point interface can
have a different MTU. But, that is not the same as a (single) multi-access
interface with multiple neirghbors.

Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Oct 2018 08:47:45 +1300

Hi Christian,

On 2018-10-18 21:11, Christian Huitema wrote:
> level PING with variable length padding, and with the DF bit set. 

How do they do it for IPv6, where there is no DF bit?

    Brian

> The

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 21:48:08 +0200

Fred,

> interface with multiple neirghbors.

Single multi-access interface with multiple neighbours. Each neighbour represented as an adjacency in the FIB.

Ole

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 21:55:33 +0200

> On 18 Oct 2018, at 21:47, Brian E Carpenter <[email-masked]> wrote:
> 

In IPv6 the DF bit is always set!

Ole

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 21:03:00 +0000

Ole,

> Sent: Thursday, October 18, 2018 12:48 PM
> Cc: Mikael Abrahamsson <[email-masked]>; 6man WG <[email-masked]>

What is the MTU of the multi-access interface itself (e.g., "eth0")? It
must be no smaller than the maximum MTU of the neighbors, right?

Fred

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 23:19:33 +0200

Fred,

> must be no smaller than the maximum MTU of the neighbors, right?

The only must from a forwarding perspective is that the frame must be no larger than what the NIC can support.

Currently the per-protocol adjacency MTUs are inherited from the per-protocol interface MTU.
But from an implementation perspective, that’s just because there is no other MTU information about that neighbor.

https://github.com/FDio/vpp/blob/master/src/vnet/ip/ip6_forward.c#L1878
https://github.com/FDio/vpp/blob/master/src/vnet/adj/rewrite.c#L103
https://github.com/FDio/vpp/blob/master/src/vnet/interface_funcs.h#L288

Cheers,
Ole

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 21:20:04 +0000

Hi Tom,

> Sent: Thursday, October 18, 2018 12:01 PM
> To: Ole Troan <[email-masked]>

Larger MTUs result in fewer interrupts and less protocol headers.
For a 64K UDP datagram, if it can be sent in one piece that is a single
UDP/IP header and a single packet. If it needs to be broken down
into 1500 chunks, then there are 40-some packets and around 1K
of extra headers.

But, how large is the largest MTU we should aim for? 9K? 64K?
Something larger? 64K seems to fit IP sizing parameters without
having to resort to IPv6 jumbograms.

Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 21:28:24 +0000

Ole,

> Sent: Thursday, October 18, 2018 2:20 PM

How does it do link-scoped multicast?

Fred

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 23:35:47 +0200

> On 18 Oct 2018, at 23:28, Templin (US), Fred L <[email-masked]> wrote:
> 
> Ole,
> 

Limited to the per-protocol per-interface MTU. 

Which in this case would have to be the smallest common MTU supported, and the per-neighbour MTU could go up to maximum supported by neighbour and NIC. 
Ole

---

From: Bob Hinden <[email-masked]>
Date: Thu, 18 Oct 2018 14:38:17 -0700

> On Oct 18, 2018, at 2:19 AM, Nick Hilliard <[email-masked]> wrote:
> 

Sure, we can add that to the next version.

Bob

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 21:40:32 +0000

Hi Ole,

> Sent: Thursday, October 18, 2018 2:36 PM

That is nice, but it does not square with RFC3819. What you
are describing would have to update or obsolete RFC3819.

Thanks - Fred

---

From: Ole Troan <[email-masked]>
Date: Thu, 18 Oct 2018 23:59:15 +0200

> On 18 Oct 2018, at 23:40, Templin (US), Fred L <[email-masked]> wrote:
> 
> Hi Ole,
> 

Fred,
what we are exploring here is changing the model where a link has a fixed MTU. 
(Cause at the datalink layer it doesn’t. )

Cheers 
Ole

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 18 Oct 2018 22:27:47 +0000

Hi Ole,

> Sent: Thursday, October 18, 2018 2:59 PM

Here is what RFC4861 says about variable MTU links:

     variable MTU   - Neighbor Discovery allows routers to specify an
                      MTU for the link, which all nodes then use.  All
                      nodes on a link must use the same MTU (or Maximum
                      Receive Unit) in order for multicast to work
                      properly.  Otherwise, when multicasting, a sender,
                      which can not know which nodes will receive the
                      packet, could not determine a minimum packet size
                      that all receivers can process (or Maximum Receive
                      Unit).

Don't get me wrong - I don't mind seeing a new link model where
the interface MTU is set to the *minimum* MTU among all nodes
on the link for the purpose of multicast, and then use per-nbr
MTUs for larger MTUs for specific neighbors.

But, this "new" link model would require updating at least RFC3918,
RFC4861 and probably several others. It also requires changes to the
IP protocol stack such as the code you sent me in your last message.

Thanks - Fred

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 19 Oct 2018 14:33:07 +1300

> In IPv6 the DF bit is always set!

But if the source chooses to include a fragment header, DF is clear.
And whether it does that depends on O/S-dependent socket options
IPV6_PMTUDISC_DO, IPV6_PMTUDISC_PROBE, IPV6_DONTFRAG which behave
differently in different stacks (e.g. the exact version of Windows 10
matters). So I really would be interested in how QUIC does it.

Please enjoy the attached Python3 code.

    Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 19 Oct 2018 06:12:03 +0200 (CEST)

> But, you can't send larger than the interface MTU.

Correct.

> RFC3819 is the BCP. So, you would either have to update or obsolete it.

I don't have a problem with that.

---

From: Ole Troan <[email-masked]>
Date: Fri, 19 Oct 2018 09:33:55 +0200

Fred,

> IP protocol stack such as the code you sent me in your last message.

Yes, agree with all that.

If there is enough value in doing all that I’m not sure. One would think that in environments where a higher MTU mattered one would be able to configure all nodes on the link with a higher MTU…

Best regards,
Ole

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 19 Oct 2018 10:11:56 +0200

> If there is enough value in doing all that I’m not sure. One would think that in environments where a higher MTU mattered one would be able to configure all nodes on the link with a higher MTU…

You will never reach a tipping point with that thinking.

Deployment must be incremental!

Grüße, Carsten

---

From: Christian Huitema <[email-masked]>
Date: Fri, 19 Oct 2018 10:16:10 +0200

> On Oct 18, 2018, at 9:47 PM, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi Christian,
> 

They do the equivalent : use a socket option to tell the kernel to not insert a fragmentation header. 

--Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Fri, 19 Oct 2018 10:21:16 +0200

> On Oct 18, 2018, at 9:55 PM, Ole Troan <[email-masked]> wrote:
> 
> 
> 

You wish, but that's not what Linux or Windows does. The kernel code may attempt its own PMTU discovery, and will do end to end fragmentation inserting a fragmentation header if the application sends a UDP packet larger than the kernel's idea of PMTU.  If the application like QUIC doesn't want that, it has to explicitly disable the kernel function.

-- Christian Huitema

---

From: Ole Troan <[email-masked]>
Date: Fri, 19 Oct 2018 11:04:55 +0200

Carsten,

> Deployment must be incremental!

Good. I will count that as one voice in favour of developing link support for per-neighbour MTU.

Cheers,
Ole

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 19 Oct 2018 11:16:42 +0200

> Good. I will count that as one voice in favour of developing link support for per-neighbour MTU.

Indeed.

We have had trouble doing support for pairwise capabilities in ND in a number of places.

See https://tools.ietf.org/html/draft-bormann-rohc-over-802-02
for another example where that would have been useful.

(RFC 7400 6CIO provides a capability indication mechanism for 6LoWPAN-ND, but we haven’t ported that to ND classic yet.)

Maybe focusing the MTU > 1500 deployment issue on IPv6 and ND would make it cleaner…

Grüße, Carsten

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 19 Oct 2018 11:25:45 +0200

Mikael Abrahamsson wrote on 18/10/2018 18:51:
>> underlying complexity associated with configuring them goes from O(1) 
>> to O(n^2) where n is the number of devices talking to each other, 
>> simplicity, and any decision to move away from a simpler system needs 
>> to be taken in the context of risk and long term cost/benefit.
> 

oh sure - this will all be negotiated at a layer lower than most end 
users will be aware.  It's more an issue of what happens when things 
break.  Adding complexity has a cost.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 19 Oct 2018 12:40:19 +0200

> In IPv6 the DF bit is always set!

this is largely why this draft isn't a good idea, or at least why it 
isn't a good idea right now.  The basis is simple enough: if there is a 
large path MTU between A and B, then it makes sense to use that and 
benefit from the performance advantages of larger MTUs in places where 
it matters.

The problem relates to the failures modes: path MTUs between A and B are 
not guaranteed to remain the same.  When they change, what happens?  UDP 
will be blackholed and TCP will be blackholed unless MSS is adjusted.

 From an operational point of view, path mtus change all the time.  In 
the ipv4 world, this is a minor annoyance because you end up with 
fragments, but life continues on.  In the IPv6 world, things will break. 
  We know what the failure modes are because in the operational worls we 
occasionally see L2 paths

So, I'm not against the idea of this draft, but we need to fix ipv6's 
lack of in-path fragmentation first.  Once this has been resolved 
properly, then the next step is to build the tools to handle path mtu 
negotiation

Nick

---

From: Tim Chown <[email-masked]>
Date: Mon, 22 Oct 2018 13:30:39 +0000

> On 19 Oct 2018, at 09:11, Carsten Bormann <[email-masked]> wrote:
> 

The incremental thinking in the networks Dale mentioned earlier comes from the separate treatment of "business" and research/science traffic, rather than incremental deployment within specific links.  Support for 9K MTUs is widespread on national research and education networks, and for the connectivity between them.  Thus connected universities or research organisations can deploy larger MTUs at their site or campus topological edge, offering their compute and storage nodes the performance advantages of the larger MTUs for data exchanges with other sites around the world, while still leaving their regular campus traffic and internal user networks running as they are. 

As an aside, some of those communities are also pushing ahead quite aggressively with IPv6, not least the storage systems that exchange data from Large Hadron Collider experiments run at CERN; the aim there has been to dual-stack all Tier 1 and Tier 2 storage sites so that new IPv6-only compute clusters can then draw down data without the need for any form of NAT/translation in the network.

Tim

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Oct 2018 19:59:13 +0200

Folks,

A couple of comments/questions, as I read the I-D:

1) What's the motivation of this I-D over, say, traditional PMTUD? While
skimming through the doc, it might save a few RTTs in cases where
there's a decreasing MTU (on the way to the destination). However, given
that the proposal is targeted at datacenters, and that PMTUD information
is not updated frequently, the advantage in this respect would be
negligible.

2) Looks like the option should have a checksum? In PMTUD, packets do
contain a checksum. However, IPv6 packets do not.

(may send more comments as I finish reading the doc).

Thanks,
Fernando

-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: G Fairhurst <[email-masked]>
Date: Tue, 23 Oct 2018 19:10:10 +0100

> is not updated frequently, the advantage in this respect would be
Well path characteristics are getting pretty hard to cache with ECMP 
often deployed and not knowing a priori which path a packet flow may 
take, so I'd expect each flow has to find out for itself if it can use a 
bigger MTU.
> (may send more comments as I finish reading the doc).
I'm not sure what a checksum would buy... to me the value returned is a 
hint. It's not critcial if it is wrong/corrupted, but it would eliminate 
the benefit.
>>> *From: [email-masked]<mailto:[email-masked]>
>>> *To: *"Godred Fairhurst"<[email-masked]
>>> <mailto:[email-masked]>>, "Robert Hinden"<[email-masked]
>>> <[email-masked]<mailto:[email-masked]>>, "Gorry
>>> Fairhurst"<[email-masked]<mailto:[email-masked]>>
>>>
>>>

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Oct 2018 20:03:07 +0200

> Maybe it’s time to pull the old draft out again.  The hop-by-hop option problem of course hasn’t gone away.

IPv6 EHs are not readily available on the public IPv6 Internet. See
RFC7872. While not included in RFC7872, my measurements indicate that
the same applies even to IPsec EHs. :(

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Oct 2018 20:06:18 +0200

> reasons why this wouldn't work, which should start me to be concerned?

Are you referring to some specific network scenario, or about the
general case, where this protocol would operate on the public Internet?

Cheers,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Oct 2018 20:10:55 +0200

> any and all protocol consequences.

Never underestimate resurrection :) :
https://gcn.com/Blogs/CyberEye/2013/08/Microsoft-patch-ping-of-death-IPv6.aspx

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 23 Oct 2018 20:26:04 +0200

> take, so I'd expect each flow has to find out for itself if it can use a

mm.. but.. what's the difference between what you are proposing, and
traditional PMTUD, in this respect?

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Tue, 23 Oct 2018 18:57:27 -0700

> the same applies even to IPsec EHs. :(

Fernando,

If I'm reading RFC7872 correctly, it is more accurate to say that the
drop rates for EH vary between 12% and 55% depending on type and
destination. We cannot say that EH always works on the Internet, but
neither is it correct to say that EH never works on the Internet.
Success rate is somewhere in between.

While the data in RFC7872 is objective, the conclusions being drawn
are not. I think there are two possible conclusions 1) Success rate of
EH is abysmal, EH is a failure and should be deprecated, and the
would-be contents of EH need to be moved to other layers somehow 2) EH
support is less than desirable, but this is a current condition that
may be improve over time with new deployments, protocol
clarifications, definition of some "useful" options. IMO, until EH is
officially deprecated, it is entirely appropriate to define new uses
such as the Path MTU Option here. Since we know up front that the
option might not always be viable on the Internet, the mechanism will
need a fallback; but the need for a fallback is hardly unique: IPv6
falls back  IPv4 via happy eyeballs, UDP falls back to TCP when
firewalls block UDP, etc.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Wed, 24 Oct 2018 05:14:44 +0200

> Success rate is somewhere in between.

Can you depend on something that fails in 10% (or more) of cases?

> would-be contents of EH need to be moved to other layers somehow

Tags aside, the reality is that if you need the the would-be contents to
be reliable, you'd put them else where. Not that I like it... but
whatever my personal taste is, it doesn't solve problems by itself.

(see what happened to ip options... the ipv6 eh case is worse).

> 2) EH
> clarifications, definition of some "useful" options. 

Yes, it could improve. One never knows. Me, my gut feeling is that's
unlikely.

> IMO, until EH is
> such as the Path MTU Option here.

No matter how EHs are flagged, we're doing engineering. It works, or it
doesn't. (as unfortunate as that may be).

To me, it all looks like they are reliable in controlled domains (within
your network, where you control everything), but not elsewhere (public
Internet)

> Since we know up front that the
> need a fallback;

If the fall-back "just works" why not keep it simple?

> but the need for a fallback is hardly unique: IPv6

I'd compare such a fall-back with e.g. retrying TCP connections without
the ECN bits set. When doing the fall-back, you lose the benefits of
ECN. - but I would expect that in most EH cases, loosig the
functionality is not a viable option. e.g., what would the fall-back be
for IPsec packets that get blackholed?   (I tel you *my* workaround: I
tunel over TLS over a transport protocol.. don't even bother to try to
do IPsec in the general case)

Just to make my point clear:not that I like the situation. BUt
pretending it doesn't exist doesn't make it go away.

Cheers,

---

From: Havard Eidnes <[email-masked]>
Date: Wed, 24 Oct 2018 09:34:38 +0200 (CEST)

> https://gcn.com/Blogs/CyberEye/2013/08/Microsoft-patch-ping-of-death-IPv6.aspx

"Nice" :(

- Håvard

---

From: Tom Herbert <[email-masked]>
Date: Wed, 24 Oct 2018 08:45:56 -0700

> Can you depend on something that fails in 10% (or more) of cases?
Well, aren't we also trying to depend on IPv6 itself which still has a
pretty high failure rate on the Internet? If we're restricted to only
use protocols that we can absolutely depend on to work then the only
choice is to use plain TCP over plain IPv4.

Also, I think "depend" is an interesting word in this context. By
definition the Path MTU option is an *option*, it cannot be required
for basic communications. No one can depend on it. It is an
optimization though, and as such does need to be used X% of the time
before it can be claimed to have value.

> whatever my personal taste is, it doesn't solve problems by itself.
The problem there is that no one has yet to define the alternative
that would work any better.

> doesn't. (as unfortunate as that may be).
I disagree that it has to be a global binary choice. Happy eyeballs
has shown how to deal with a protocol that has greater than zero, but
less than 100% support.

> your network, where you control everything), but not elsewhere (public
That's not clear at all. For a small domain this might be plausible,
but for large sprawling domains like the big DC operators that use
multiple vendors it may or may not be true. Besides that, if the plan
is to only use features like EH in a controlled domain, then we need
to have an implementation at host to determine whether the destination
is in the controlled domain-- that is not necessarily easy to do. A
common solution should consider whether the feature is viable to each
particular destination.

> If the fall-back "just works" why not keep it simple?
Because fallback means we don't get the advantages of the feature.

> do IPsec in the general case)
I think IPsec is an extreme. For other use cases of EH, particularly
DO and HBH, like Path MTU option, OAM, FAST the fallback is not to use
the option. Communication can proceed but advantages is lost possibly
resulting in a degraded mode of service relative to using the option.

> pretending it doesn't exist doesn't make it go away.

Agreed.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Thu, 25 Oct 2018 00:00:04 +0200

Hello, Tom,

On 10/24/18 5:45 PM, Tom Herbert wrote:
> pretty high failure rate on the Internet? 

I haven't measured IPv6 failure rate, I should say.

> If we're restricted to only

I guess that there are things that are likely to be workable, and others
that aren't. The structure of EHs doesn't seem to be friendly wth modern
router architectures
(https://iepg.org/2015-11-01-ietf94/IEPG-RouterArchitecture-jgs.pdf), so
there's not much of an indication that things will move in a different
direction.

> for basic communications. No one can depend on it.

Either you have a reliable PMTUD, or you prepare to send1280-byte
packets (or smaller). So in such case the algorithm should be "the MTU
is 1280. Upon receipt of a PTMU option, rase it to, at most, the MTU of
your outgoing interface". Otherwise traffic can get blackholed and you
get a DoS.

> that would work any better.

Currently (SR aside), you need EHs from fragmentation and IPsec. For
fragmentation, it remains to be researched how things are working out
(maybe fall back to v4 for the DNS,.. but I don't know). For IPsec...
well, you probably avoid native IPsec traffic as you did for v4, anyway.

Put another way, looks like nobody   looked for an alternative because
nobody "needed" it.

> has shown how to deal with a protocol that has greater than zero, but

It's a different thing. Happy eyeballs essentially deal with multiple
choices where all of them are equivalent. You just need one to work.
Happy Eyeballs wouldn't work if you tried to use it when acecss stes
behind a firewall that filters port 80 and 443.

If, say, IPsec fails, you can't just "disable ipsec". or "well, lets try
a TLS tunnel".

> common solution should consider whether the feature is viable to each

So, just as a sample scenario: how ould you workaround non-working IPse
(due to filtering of IPsec EHs?)

> Because fallback means we don't get the advantages of the feature.

That means that the possible outcome, for the pmtu option , is that you
end up using 1280 instead of some larger PMTU.  (The pmtu option is
subject to blackholes... hence the fall back needs to be the minimum
IPv6 MTU, or PLPMTUD).

> resulting in a degraded mode of service relative to using the option.

For some of these cases, I'd agree. But for PMTUD, it's tricky (see above).

Cheers,

---

From: Tom Herbert <[email-masked]>
Date: Wed, 24 Oct 2018 16:20:20 -0700

> there's not much of an indication that things will move in a different

I believe the emergence of programmable devices, like those with P4,
change that. RFC8200 also allows devices to skip over HBH, this should
be trivial code in P4.

> Put another way, looks like nobody   looked for an alternative because

There also been a number of proposals to get OAM data into the packet.
Suggestions have included stealing bits from flow label, diffserv
bits, or putting them in TCP. None of these have been shown to be
viable. And of course there was SPUD proposal, which in the end would
have more issues than getting EH to work.

> (due to filtering of IPsec EHs?)

I'd probably tunnel it over UDP (GUE to be precise). If UDP doesn't
work because it's blocked by a firewall, then I'd tunnel over TCP (GTE
described in https://tools.ietf.org/html/draft-herbert-tsvwg-gte-00).

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 25 Oct 2018 14:13:47 +1300

> change that. RFC8200 also allows devices to skip over HBH, this should

Right, implementers with network processors have always liked the IPv6
header much more than people using, e.g., FPGAs. But unfortunately
the less flexible devices aren't going to vanish overnight.

   Brian