Subject: Working Group Last Call for draft-ietf-6man-mtu-option-06

From: [email-masked]
Date: Mon, 9 Aug 2021 10:41:11 +0200

This message starts a new two week 6MAN Working Group Last Call on advancing:

   Title           : IPv6 Minimum Path MTU Hop-by-Hop Option
   Authors         : Robert M. Hinden
                     Godred Fairhurst
   Filename        : draft-ietf-6man-mtu-option-06.txt
   Pages           : 19
   Date            : 2021-08-07

https://datatracker.ietf.org/doc/draft-ietf-6man-mtu-option/

as an Experimental Track document.

Substantive comments and statements of support for publishing this document should be directed to the [email-masked] mailing list. Editorial suggestions can be sent to the author.

This last call will end on 23 August 2021.

Thanks,
Ole

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 9 Aug 2021 13:53:06 +0100

[email-masked] wrote on 09/08/2021 09:41:
> Substantive comments and statements of support for publishing this
> document should be directed to the [email-masked] mailing list.
> Editorial suggestions can be sent to the author.

zooming out on this, it looks like this proposal is suggesting that each 
forwarding device processes and updates the hbh header (where 
configured) on a per-packet basis.  This means that the practical 
reliability of this mechanism in a mixed-mtu end-to-end configuration 
depends on the ability of each intermediate device to process the hbh 
header.

draft-hinden-6man-hbh-processing requires that each forwarding device 
processes the first option in the fast path.  The mtu-option draft isn't 
going to be viable on real networks unless the latest version of the 
hbh-processing draft is widely implemented.  Given that rfc8200 relaxed 
EH processing requirements (due to the fact that the previous spec was 
so widely ignored), the likelihood of this being tightened up on 
production equipment seems remote.

The overall aspiration is clear, but given the long history of practical 
problems relating to EH processing on intermediate nodes, I don't think 
that this draft is realistic enough to move forward.

Nick

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 9 Aug 2021 15:32:49 +0100

On 09/08/2021 13:53, Nick Hilliard wrote:
> zooming out on this, it looks like this proposal is suggesting that 
> each forwarding device processes and updates the hbh header (where 
> processes the first option in the fast path.  The mtu-option draft 
> isn't going to be viable on real networks unless the latest version of 
> the hbh-processing draft is widely implemented.  Given that rfc8200 
> relaxed EH processing requirements (due to the fact that the previous 
> spec was so widely ignored), the likelihood of this being tightened up 
> on production equipment seems remote.
>
> The overall aspiration is clear, but given the long history of 
> practical problems relating to EH processing on intermediate nodes, I 
> don't think that this draft is realistic enough to move forward.
>
> Nick

Thanks Nick,

What you say is mostly what I would expect: If there is no support, the 
method does not have any benefit.

There are some important differences in this approach compared to some 
previous suggested uses of HBH headers:

* One difference is the marginal impact of packet loss on performance. 
We'd suggest that unless there is some prior knowledge about the path, 
the HBH header is only included on an infrequently sent "probe" packet. 
That is, a packet that does not carry data (i.e., does not predjudice 
protocol operation/performance if lost). DPLPMTUD will often work best 
using such packets to test a path. Of course, if the path does support 
the method, an endpoint might get some useful information!

* An important difference is *how* this information is handled by the 
endpoint/host. Normally when "probing" a path to discover the largest 
supported PMTU, the endpoint has no useful information to seed the 
search algorithm, so this might mean it takes quite a few rounds of 
probes to discover a path with a larger PMTU. This method helps seed 
that search, but we know might ultimately fail to increase the PMTU for 
the connection: importantly if there are undetected devices on path that 
do not update the HBH value, these might result in larger unusuccessful 
probe packets, but stiill this does not blackhole the actual communication.

So here is one case we see as easier to deploy than others:,It is a 
datacenter deployment where routers and switches are configured to 
support jumbo frames... getting that configuration size consistent 
across a DC can be a pain, and this might help avoid unhelpful 
blackholing when sizes do not match.

Does any of this help explain why we think this might be of value?

Gorry

---

From: Tom Herbert <[email-masked]>
Date: Mon, 9 Aug 2021 07:49:04 -0700

> production equipment seems remote.

Nick,

You seem to be making a broad generalization that all router
implementations don't properly handle HbH. I don't believe that's true. In
a limited domain an operator controls enough to ensure HbH is usable, and
even on the public Internet the data shows not all paths have a problem.
Like any other new protocol there's no expectation that the whole Internet
will support it from day one (how many years did it take for IPv6 to get
any traction ? :-) ). Deployment of protocols takes time, and
implementations are free to evolve to support new protocols.

> that this draft is realistic enough to move forward.

It's Experimental anyway, and I believe will be quite useful at least in
some networks and support moving the draft forward.

Tom

> Nick
>
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 Aug 2021 09:52:18 +1200

I can't see any objection to this as Experimental, although I suspect it will have difficulty being deployed on the paths where it is most needed.

However, I see two gaps in the draft:

1. How and when will the experiment be evaluated?

2. An "Implementation Status" section would be very helpful. 

Regards
   Brian Carpenter

---

From: Jen Linkova <[email-masked]>
Date: Tue, 10 Aug 2021 10:03:17 +1000

> I can't see any objection to this as Experimental, although I suspect it will have difficulty being deployed on the paths where it is most needed.

That might be very true, though I don't think we can expect to win a
lottery if we don't even try to buy a ticket...

> 1. How and when will the experiment be evaluated?

IMHO it would be really hard to predict right now... What are the
possible scenarios?
- the draft is published but is not widely deployed. Then it just dies quietly.
- it achieves some limited deployment, not enough for anyone spending
efforts on standardizing it.
- it gets deployed widely enough that the community would like to see
the option standardized. In that case we shall evaluate the success of
the experiment when time comes. The criteria might be different from
what we can imagine now.

> 2. An "Implementation Status" section would be very helpful.

+1.

-- 
SY, Jen Linkova aka Furry

---

From: Jen Linkova <[email-masked]>
Date: Tue, 10 Aug 2021 10:04:10 +1000

I reviewed the -06 version of the draft, I think it's a useful
experiment and the draft is in the good shape, so IMHO we shall
advance it.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 10 Aug 2021 10:45:02 +0100

Tom Herbert wrote on 09/08/2021 15:49:
> implementations don't properly handle HbH. I don't believe that's true. 

Nope, never suggested that :-)

Please look at previous ietf docs about the reliability levels 
associated with sending packets with EHs.  What I will assert as a 
generalisation is that any protocol which has endemic packet loss levels 
above very low single digit percentages is effectively unusable in 
production.

Nick

---

From: [email-masked]
Date: Tue, 10 Aug 2021 12:25:22 +0200

Nick,

>> You seem to be making a broad generalization that all router implementations don't properly handle HbH. I don't believe that's true.
> 
> Please look at previous ietf docs about the reliability levels associated with sending packets with EHs.  What I will assert as a generalisation is that any protocol which has endemic packet loss levels above very low single digit percentages is effectively unusable in production.

It's a two way street. And the networks may adapt to application behaviour.
If we follow your line of argument to the extreme, you'd be stuck with IPv4 UDP port 53 and TCP port 443...

This is the first HBH option defined that may have some usefulness across the Internet.
It's also only used as a part of the heuristics a host/application needs to use to find the path MTU.

Given that the IETF generally is poor at predicting what will make a sucessful protocol or not...what's the harm in allowing this a chance?

Best regards,
Ole

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 10 Aug 2021 12:05:52 +0100

Gorry,

ok, there are bunch of things here.

In several important respects, the mtu-option draft can be viewed as a 
low-level workaround protocol for in-band detection of network misconfig 
and breakage.  For sure there are valid reasons to want to do this sort 
of thing in-flight, but there are costs to handling this at the protocol 
level: one of these is general complexity (see the dns camel 
discussion); another is reliability (see rfc7872); a third is that 
network misconfig / breakage is often better handled by reactive 
monitoring rather than pushing the complexity into the protocol - 
particularly in this case because it's being pushed down to the bottom 
of the protocol stack.

Zooming out in a different direction, the two use cases that you suggest 
are DC deployment and general internet use.

For general internet use, rfc7872 suggests deleterious levels of loss 
for packets which have extension headers attached.  No doubt these 
numbers have changed in the last several years but there is no reason to 
think that they have changed by the multiple orders of magnitude which 
would make the addition of a new (and pervasive) HBH header viable as a 
general proposition.

Regarding datacentre config, if your DC network config is not properly 
orchestrated, then to be completely blunt, you have bigger problems.  It 
is true that there are going to be plenty of cases of smaller mixed kit 
networks / organically-grown installations / etc where you could argue 
that maybe mtu-option could help. I'd argue that if your network was 
configured in such a way that mtu-option was going to make a substantial 
difference, your network was pretty broken to start with.

The value proposition of large DC installations is scale and economics 
and at that point, you either have an orchestrated network or you're not 
in the game.  The value in this scenario is breakage workaround and I 
would argue that reactive monitoring is probably a better overall 
solution, in the sense that it's non-pervasive and can be carefully 
targeted.

On the specific points you brought up:

> the method, an endpoint might get some useful information!
You're going to have prior knowledge about the path in the DC context + 
the ability to monitor proactively.

> probe packets, but stiill this does not blackhole the actual communication.

running code would help in this situation.

> Does any of this help explain why we think this might be of value?

I get your argument, but I'd suggest that overall value proposition 
(i.e. cost-to-return ratio) of this protocol is not at all clear.

You're going to have to forgive me for saying it, but you're proposing 
to attach a protocol add-on option with the intention of making the 
protocol transmission more reliable in one respect, where the add-on 
itself is known to cause serious transmission reliability problems in 
other respects. This juxtaposition of expectations is jarring :-)

Nick

---

From: "Pengshuping (Peng Shuping)" <[email-masked]>
Date: Tue, 10 Aug 2021 13:43:05 +0000

Hi,

It definitely deserves the chance. I appreciate the authors' efforts. We have to start from somewhere (不积跬步，无以至千里).

Therefore, I support to progress this work further.

Thank you!

Best regards,
Shuping

发件人：otroan <[email-masked]>
收件人：Nick Hilliard <[email-masked]>
抄 送：6man WG <[email-masked]>;6man Chairs <[email-masked]>
时 间：2021-08-10 18:26:18
主 题：Re: Working Group Last Call for draft-ietf-6man-mtu-option-06

Nick,

Best regards,
Ole

---

From: Tom Herbert <[email-masked]>
Date: Tue, 10 Aug 2021 08:25:06 -0700

Nick,

I don't believe your generalization holds true. There are counter
examples in deployments effectively using EHs in production. In fact,
For instance, I know of at least one hyperscaler that is not only
using EH, but one EH in production is Fragment Header (fragmentation
is yet another technique that some in IETF like to bash as not being
usable in production).

Tom

---

From: Tom Herbert <[email-masked]>
Date: Tue, 10 Aug 2021 08:35:40 -0700

Nick,

RFC7872 is more than five years old, not exactly what I would call
current data. The effects of the relaxed requirements in RFC8200 that
intermediate nodes can ignore HbH instead of processing them have yet
to be measured. Furthermore, there is also ongoing work to limit
extension headers, draft-hinden-6man-hbh-processing and
draft-herbert-6man-eh-limits, to make them feasible for (hardware)
implementations. IMO we shouldn't restrict evolution of IETF protocols
based on one limited view snapshot in time, especially before even
trying to address the issues that such a snapshot raises.

Tom

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 10 Aug 2021 17:07:09 +0100

Tom Herbert wrote on 10/08/2021 16:25:
> I don't believe your generalization holds true.

My generalisation is that double digit % packet loss is deleterious on 
production networks. It's a bit puzzling not to believe this, if that's 
what you meant.

Otherwise, rfc7872 is an output of v6ops.  If you have an issue with the 
document, then by all means raise it in v6ops.

> There are counter

I've totally no doubt about it.  rfc7872 refers to generalised public 
Internet measurements and you're referring to a hyperscaler DC network 
where the capabilities and limitations of the networking kit + control 
stack will be well known and understood by the organisation.

As a side note, which of the hyperscalers runs COTS networking kit with 
a COTS control stack for their dc networks?  Or uses the cli without 
orchestration to configure them?  These networks are completely 
different beasts to the average network out there, so quoting them as 
examples is only useful when you're talking about other hyperscaler dc 
networks.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 10 Aug 2021 17:08:02 +0100

Tom Herbert wrote on 10/08/2021 16:35:
> current data.

if you can show that the % failure figures in rfc7872 have dropped by 2+ 
orders of magnitude on the public internet in the last 5 years, then I'd 
be arguing that mtu-option was in a great place to head down the 
experimental track.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Tue, 10 Aug 2021 09:39:32 -0700

Nick,

I very much disagree with that. Hyperscalers, or more generally
limited domains, are citizens on the Internet just as much as any
random provider network. They use the same IETF protocols and
standards. Not only that, IETF has developed protocols that only work
in these limited domains, a good example being SRv6 which also happens
to use extension headers. Try SRv6 packets into the Internet and
you'll likely see extremely high loss rates, but that hardly means the
protocol isn't useful and shouldn't be abandoned. The same thing
applies to the path-MTU option, if someone finds it useful in their
deployment or just wants to experiment with it, why deny them that
opportunity?

Tom

---

From: [email-masked]
Date: Tue, 10 Aug 2021 19:00:44 +0200

Nick,

> if you can show that the % failure figures in rfc7872 have dropped by 2+ orders of magnitude on the public internet in the last 5 years, then I'd be arguing that mtu-option was in a great place to head down the experimental track.

Prior to this work there were no HBH options that had Internet wide applicability.
The combination of no specified HBH options in existence and the existence of some "risky" HBH processing implementations meant that filtering HBH in the network was reasonably sensible.

The existence of this draft changes the equation.
Will that make HBH deployable in the Internet. Who knows.
I see your point "this work is futile because it's not deployable".
But what's the worst that can happen?
We get some more advanced MTU probing algorithms in endpoints and a few router vendors burn the MTU option into silicon?

If the experiment failed, would the existence of the experiment cause harm?

Best regards,
Ole

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 10 Aug 2021 15:42:09 -0400

+1

I think this draft this MTU endpoint probing  will really help the overall
viability of the use or HBH options for IOAM and other use cases deployable
on the internet.

As Ole mentioned I don’t see a down side to adoption.

Kind Regards

Gyan

On Tue, Aug 10, 2021 at 1:01 PM <[email-masked]> wrote:

> orders of magnitude on the public internet in the last 5 years, then I'd be
> arguing that mtu-option was in a great place to head down the experimental
> track.
>
> The combination of no specified HBH options in existence and the existence
> of some "risky" HBH processing implementations meant that filtering HBH in
> the network was reasonably sensible.
>
> We get some more advanced MTU probing algorithms in endpoints and a few
> router vendors burn the MTU option into silicon?
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 10 Aug 2021 22:19:32 +0100

> I see your point "this work is futile because it's not deployable".

more fundamentally, this proposes to create a protocol to marginally 
increase performance and reliability in one direction which depends on a 
component which is known to be highly unreliable in another.  This seems 
ill-advised.

> But what's the worst that can happen?

The camel problem + arbitrary connectivity breakage if the protocol is 
enabled on hosts transmitting data over the wider internet.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Tue, 10 Aug 2021 15:39:19 -0700

Nick,

That situation would be no different than any other case where we've
deployed a protocol that wasn't on the currently "approved" protocols
list of some router vendors and we had to deal with that, e.g. ICMP,
fragmentation, any transport protocol other than TCP, QUIC, TCP fast
open, and even IPv6 itself. If it's a requirement that a new protocol
needs to have 100% reachability in the Internet from day one, we'd
never be able to specify any new protocols and the ossification of the
Internet would be complete.

Tom

---

From: "Dongjie (Jimmy)" <[email-masked]>
Date: Wed, 11 Aug 2021 03:38:22 +0000

Hi, 

I've read the latest version of this draft, and support its publication as an Experimental track document. It is a useful feature and could promote the implementation and deployment of IPv6 HBH. 

Best regards,
Jie

> -----Original Message-----
> From: ipv6 [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Monday, August 9, 2021 4:41 PM
> To: 6man WG <[email-masked]>
> Cc: Gorry (erg) <[email-masked]>; 6man Chairs <[email-masked]>
> Subject: Working Group Last Call for draft-ietf-6man-mtu-option-06
> 
> Substantive comments and statements of support for publishing this document
> should be directed to the [email-masked] mailing list. Editorial suggestions can be
> sent to the author.
>

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 11 Aug 2021 15:59:06 +0100

Tom Herbert wrote on 10/08/2021 23:39:
> open, and even IPv6 itself.

Tom,

you make a good point here - if it were an entirely new protocol, that 
would be fine because there wouldn't be any regression associated with 
deploying it. This draft is materially different in the sense that 
you're taking a protocol suite which is known to work reasonably well 
and bolting an add-on to the side which is known to cause serious 
problems, and then hoping that in time, general hbh support will improve 
to the point that the overall impact will be a net improvement.  The 
reality is that it creates a high operational support cost. How can you 
disable / debug it? Is it implemented as a socket option? As a 
kernel-wide default?  Both?  On a src/dst/proto/service basis?  How are 
the failures logged?  Who do you contact when hbh-enabled packets are 
dropped at some other location on the internet, etc?

Nick