Subject: Re: draft-van-beijnum-multi-mtu-05.txt

From: David Lamparter <[email-masked]>
Date: Wed, 6 Apr 2016 17:18:31 +0200

On Mon, Apr 04, 2016 at 11:37:17PM +0200, Mikael Abrahamsson wrote:
> I'll be presenting 
> https://tools.ietf.org/html/draft-van-beijnum-multi-mtu-05 in 6man on 
> Wednesday. There are operational considerations here, so I encourage 
> people to drop in and/or comment on this draft.

Ok... easy comments/opinions first:
- should definitely be ICMP, not UDP.  Erik Nordmark's suggestion to
  piggyback on NUD sounds great, though admittedly I haven't thought it
  through looking for pitfalls.

- if possible (not sure on NUD interactions), this should use RFC5082
  TTL security, sending HL=255 and discarding packets !=255

- there's probably a lot of 4821 that can be incorporated by reference,
  shortening the draft by quite a bit

- Mark Townsley's comment about extension headers doesn't make sense to
  me; we're looking at a single subnet here -- do we really want to
  support a non-fully-transparent layer 2 bridge messing up the packets?

- the padding should be specified as "MUST be non-zero" and checksum
  validation a "MUST";  I've personally seen devices that zero out
  packets beyond their size limits but still switching them.

- ditch IPv4, but +1 to Mark's "MUST NOT interact/break" room comment

Now the hard bit; L2 devices messing with L2 addresses.  The device
class I have in mind here are 802.11 "client to wired" bridges.  They
connect to wifi as plain clients and have one or more LAN ports
supporting multiple devices.  802.11 restricts them to the single MAC
address their radio has, so what they do is they "MAC-NAT" everything
behind them.  (They rewrite ARP & ND and have an address table to undo
it on incoming wireless packets.)

The -05 draft currently says "Neighbors are identified by combination of
a link address and an IP version".  (NB: IP version, not IP address...)

I believe the better choice here is "Neighbors are identified by their
IPv6 link-local address".  This removes any possibility for 2 hosts to
be conflated into a single neighbor even in the presence of 'broken' L2
devices.
It also makes clear that for forwarded packets beyond the subnet, the
mechanism operates on the (host, router) pair and in no way interacts
with L3 PMTUD or holds state for any global-scope address.  The (host,
router) pair would establish the maximum possible MTU for packets taking
that route, from which PMTUD then works downwards.

(Using link-local would introduce other corner cases, particularly if
the link layer address changes, probe data isn't implicitly
auto-discarded; this should become an explicit action.  There's probably
more corners I didn't think of.)

-David

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 Apr 2016 13:12:45 -0400

> it on incoming wireless packets.)

This definitely falls into the category of stupid-layer-2 tricks to get
around a lack of a routed (homenet-style) layer-3.  sigh.  IPv4 legacy.

It seems you are saying with can deal with this problem with:

> I believe the better choice here is "Neighbors are identified by their IPv6
> link-local address".  This removes any possibility for 2 hosts to be
> conflated into a single neighbor even in the presence of 'broken' L2
> devices.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: David Lamparter <[email-masked]>
Date: Wed, 6 Apr 2016 19:43:08 +0200

> around a lack of a routed (homenet-style) layer-3.  sigh.  IPv4 legacy.

Indeed, though it's also 802.11 failing to cover bridges as plain
clients.  802.11ak (= work in progress) and non-standardised WDS (widely
available) solve the same issue, but they need AP implementation (11ak)
or even configured cooperation (WDS); thus that class of kludge products
will stay around for quite a while...

> It seems you are saying with can deal with this problem with:

Yes (sorry if I worded this poorly) - I arrived at the suggestion below
in the hope that identifying by link-locals is less prone to L2
shenanigans than identifying by link-layer address.

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 6 Apr 2016 16:55:46 -0300

>    through looking for pitfalls.
One issue compared to a dedicate probe is that a NUD (a unicast NS) of 
size X wouldn't necessarily result in a unicast NA that is also padded; 
perhaps one can add such behavior so that we can get the bidirectional 
probe in both directions.

Note that just like TCP can suppress the need for NUD (per RFC 4861) a 
TCP implementation could also indicate to the IP layer than a packet of 
size X got through, which could be used to suppress the MTU probe. That 
applies independently of what protocol is used to do the MTU probing.
>    TTL security, sending HL=255 and discarding packets !=255
Yes, it would make sense to use the TTL security approach for the probe 
protocol, whether it is UDP or NUD or something else.
>    shortening the draft by quite a bit
Are you referring to the choice of sizes to try? I haven't re-read 4861 
next to this draft.
>    support a non-fully-transparent layer 2 bridge messing up the packets?
My understanding is that the muti-mtu draft is for the local link, and 
not for end-to-end.

An observation is that if/where RFC4821 is implemented, then the 
dependency on ICMP PTB is removed. Hence addresses Mark's comment.

>    packets beyond their size limits but still switching them.
Good point.

Regards,
    Erik

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: David Lamparter <[email-masked]>
Date: Wed, 6 Apr 2016 23:20:48 +0200

> probe in both directions.

That might be (half of) a feature: if the MTU probe is a NS option, and
the target host does not support it, we still get a NA reply.  That's
better than timing out waiting.  The downside is that a NA without the
"MTU response" option doesn't mean that the host doesn't implement the
protocol -- the NA could be in response to something else.

That said, I'd expect the probability of a host ignoring a new NS option
is much lower than it filtering out unknown new ICMP (sub)types or UDP
ports.  If that reduces useless probing (or even waiting), I'm all for
it.

(As I understand it, the draft in no case suggests waiting for the
probes to come back, so it's not much of an argument.  Might still
reduce useless probe traffic?)

> Are you referring to the choice of sizes to try? I haven't re-read 4861 

I was thinking it could be possible to inherit parts of the algorithm
and corner case behaviour spec, but the draft makes the argument that
its probing can be much more aggressive since there is (almost?) no
congestion interference.  That seems a fair point, so I guess I spoke
too quickly :(

> protocol, whether it is UDP or NUD or something else.

Rereading the draft, it already says "other than 255 [...] discarded".
So my comment turns into "add explicit reference to 5082".

-David

---

From: David Lamparter <[email-masked]>
Date: Thu, 7 Apr 2016 00:04:11 +0200

> protocol -- the NA could be in response to something else.

Obvious easy way: have the "ND NODEMTU" option described in draft sec. 5
mandatory on all NAs if the host implements the protocol; receipt of any
NA without the option can then flag the sender as mtu-probe-incapable.

(Not sure if the last paragraph of sec. 9.1 already implies the option
MUST be put on all NAs and NSs?  Certainly needs better wording.)

-David

---

From: [email-masked]
Date: Thu, 7 Apr 2016 10:27:07 -0300

In general I'd prefer that we reuse existing mechanisms for path mtu discovery, and not invent new ones purely for the case of two directly connected neighbors.

Could we make this work by adding the RFC4861 MTU option to NS/NA messages, and require that PLMTUD probing is used to verify the end to end MTU.

Then the mechanism used is the same for directly connected neighbors and off-link neighbors (apart from the initial discovery).

If we were to invent new probing mechanisms, I would hope we did so under the PLMTUD (4821) umbrella.
Best regards,
Ole

> On 06 Apr 2016, at 19:04, David Lamparter <[email-masked]> wrote:
>

---

From: Philip Homburg <[email-masked]>
Date: Thu, 07 Apr 2016 15:46:31 +0200

>In general I'd prefer that we reuse existing mechanisms for path mtu 
>discovery, and not invent new ones purely for the case of two directly 
>connected neighbors.

I think making them the same is likely to be sub-optimal.

The nice thing about the current proposal is that the MTU is verified before
any application traffic is sent.

This not going to fly on the internet as a whole. There is no way that
busy DNS servers are first going to send ping traffic back to the client before
sending the DNS reply.

On the other hand, the model behind existing PMTU discovery is reactive. The
loss of a data packet triggers downward adjustment of the estimated PMTU.

In the context of ethernet jumbograms, such a reactive approach is likely to be
unattractive to operators. 

So I'd say that from a PMTU perspective directly connected neighbors are quite
different from the internet as a whole.

---

From: Michael Richardson <[email-masked]>
Date: Thu, 07 Apr 2016 12:29:57 -0400

> Could we make this work by adding the RFC4861 MTU option to NS/NA
    > messages, and require that PLMTUD probing is used to verify the end to
    > end MTU.

Yes, but plmtud can only be provided as a service to applications when they
use TCP or SCTP.  It can't find bigger UDP paths at present: that would have
to be built in to the application.  But, the kernel can provide the service
for UDP users by doing the discovery in NUD....
Basically we can run the plpmtud mechanism with NUD in the kernel.

There is another use for this MTU option:  it appears that it is sometimes
possible to run both 802.15.4-2012 (the 127 byte version) and 802.15.4g (the
2K or something byte version) on the same radios.
("802.15.4-2015" includes 802.15.4g, btw, thus the need for a year to be precise)

A 15.4g MAC ought to be able to speak to a -2012 MAC by using all of our 6lowpan
mechanisms, but two 15.4g would like to discover that they don't need to
fragment.  I had *assumed* that the RA MTU option could be used for this, and
was surprised when I read that it wasn't specified for this....

So I'm in favour of having this ND option defined; and I care not if it's a
new number or reuses the RA number.

---

From: [email-masked]
Date: Thu, 7 Apr 2016 19:20:03 -0300

Michaell,

> Basically we can run the plpmtud mechanism with NUD in the kernel.

UDP applications already need to do that today.
Isn't it a great step backwards if an application first has to detect if the peer is directly connected or not, and behave differently depending on the answer?

Path MTU is a path property. I'm reluctant to make too much out of a special case where path length is 1.

I do agree that we need to do something in ND to allow nodes capable of a higher MTU than the RA advertised default MTU to signal that between each other.

Best regards,
Ole

---

From: Michael Richardson <[email-masked]>
Date: Fri, 08 Apr 2016 10:15:17 -0400

> UDP applications already need to do that today.

Yes, that's a good point.
In v4-land, we can just send and experience fragmentation.
In v6, we can't, so most WAN applications that care about >1280 always
include fragmentation at that level regardless.  So you are right: the
applications already do it.

There is a class of application that is today running over LANs using UDP.
I recall the days of "HIPPI" which had 64K packet sizes doing stuff.
I have an ATSC TV tuner that does this, and I've seen IP audio applications
that also do the same.  Right now, I think they don't even have default
route support...  I think that those devices could benefit from the ND solution.

    > Isn't it a great step backwards if an application first has to detect
    > if the peer is directly connected or not, and behave differently
    > depending on the answer?

Yes, that's a good point. So I withdraw my suggestion... the question is
perhaps, can we come up with a slightly application-neutral way to do
plpmtud.  I suspect that SPUD already provides this.

    > I do agree that we need to do something in ND to allow nodes capable of
    > a higher MTU than the RA advertised default MTU to signal that between
    > each other.

Good.

---

From: Sowmini Varadhan <[email-masked]>
Date: Mon, 11 Apr 2016 05:18:07 -0700

On Wed, Apr 6, 2016 at 12:55 PM, Erik Nordmark <[email-masked]> wrote:

> probe in both directions.

there's also the question of what mtu to advertise in the face of
pvlans, when there is a router on a promisc port proxying the NA.

In practice, since a node typically uses a single interface to talk
to multiple onlink neighbors, and jumbo mtu etc is configured per
interface, you would end up getting metered down to the lowest
common denominator for the subnet, right?

--Sowmini

---

From: David Lamparter <[email-masked]>
Date: Thu, 14 Apr 2016 14:22:45 +0200

> discovery, and not invent new ones purely for the case of two directly

That's actually the crux of it: this isn't Path MTU discovery.  The MTU
discovered by this mechanism is the first-hop MTU for a lot of paths in
case of a router and it's going to be distinct from the final PMTU
towards a particular destination.  Plus, state on this only ever exists
for direct neighbors, not for any random internet host.

> messages, and require that PLMTUD probing is used to verify the end to

It's not end-to-end MTU...

> Then the mechanism used is the same for directly connected neighbors
> and off-link neighbors (apart from the initial discovery).

Well, it'd be possible to distinguish first-hop failures from failures
later in the path, but with a router there's rarely a direct TCP
connection to it to do PLMTUD on.

Either way this really needs at least different handling from end-to-end
MTU in order to not keep probing towards a router whenever a host
decides to talk to another destination in the internet.  Some first-hop
neighbor concept...

If probing hosts for each of their addresses can be avoided, that might
be useful too, but that's far less of an issue (and harder to do) than
not probing routers for every single internet destination.

> under the PLMTUD (4821) umbrella.

Hm.  Maybe the draft can be rewritten as extension to 4821, but quite a
few things are different with reason -- not sure this is that good of a
fit...

Cheers,

-David

---

From: David Lamparter <[email-masked]>
Date: Thu, 14 Apr 2016 14:27:26 +0200

> pvlans, when there is a router on a promisc port proxying the NA.

A system doing proxy NA is ultimately a router and should be handled as
such; i.e. if done "right" there is only one 1:1 relationship for which
first-hop MTU discovery is done.

> common denominator for the subnet, right?

No, the per-interface configuration is exactly what this draft is trying
to work around.

Regards,

-David

---

From: [email-masked]
Date: Thu, 14 Apr 2016 14:30:49 +0200

David,

> for direct neighbors, not for any random internet host.

why couldn't it be done as end-to-end MTU?

the directly connected mechanism would be just to give a hint that there might be nodes here capable of a higher MTU. then do normal MTU probing / detection.

what would be gain be to have a specific protocol to probe directly connected neighbors?

Best,
Ole

---

From: David Lamparter <[email-masked]>
Date: Thu, 14 Apr 2016 14:48:07 +0200

On Thu, Apr 14, 2016 at 02:30:49PM +0200, [email-masked] wrote:
> the directly connected mechanism would be just to give a hint that
> there might be nodes here capable of a higher MTU. then do normal MTU
> probing / detection.
> 

It can:
- work (pro-)actively and independent of higher-layer protocols,
  establishing MTU once
- contain the different logic to understand that the router's MTU is the
  upper cap for all destinations behind that router
- contain the different probing parameters (being more aggressive) that
  are appropriate for a local network
- optionally carry extra information to reuse probing results across
  different addresses that a host has on a segment

I suppose this can be achieved by using a specialized version of 4821 on
the link... something like this:

- host does NS as usual, or catches unsolicited NA
- NA contains new "MTU discovery" option with contents:
  - indication of capability of first-hop MTU discovery
  - known maximum MRU of NA's sender (i.e. upper limit)
  - link-local address(?) + TCP port number(?) for PLMTUD probing
- host establishes one or more TCP sessions to that port with the
  express and only purpose of probing MTU
  (may as well do it on an existing TCP session if one exists)
- PLMTUD switches into some "first-hop" special mode (lower timers,
  parallel operation, different values) to establish and store the value
  with proper semantics (router case)

Is that what you have in mind?

-David

---

From: [email-masked]
Date: Thu, 14 Apr 2016 15:05:11 +0200

David,

> - work (pro-)actively and independent of higher-layer protocols,

to a large extent we have failed at doing MTU discovery at the network layer.
too hard, couldn't do it, let's make it someone else's problem. ;-)
transport or application that is.

> Is that what you have in mind?

I was thinking of something much simpler.
the NS/NA MTU option triggers the stack (transport / application) to use packets up to that MTU.
the transport or application has to do normal PLMTUD.

think of it as if you had a p2p link to the neighbor and that the interface MTU was set to whatever you received in the MTU option from that neighbour.

an on-link router already advertises an MTU in an RA. I'm not sure if it would make sense if it advertised per neighbor MTU options, and how those should be interpreted.

cheers,
Ole

---

From: David Lamparter <[email-masked]>
Date: Thu, 14 Apr 2016 15:19:44 +0200

> transport or application that is.

Might be either hope or naïvety, but I don't think we've failed (or even
tried) yet at the 1-hop 0-router case :).  There are way less pesky
ICMPv6-breaking devices in the middle...

> the NS/NA MTU option triggers the stack (transport / application) to
> use packets up to that MTU.  the transport or application has to do
> normal PLMTUD.
> 
> think of it as if you had a p2p link to the neighbor and that the
> interface MTU was set to whatever you received in the MTU option from
> that neighbour.

I think the failure modes of that are way worse than with a dedicated
first-hop approach, especially for routers (interference from congestion
further down) but also for hosts (no parallel probing, and still need
special-casing to get more aggressive behaviour).

(cont'd below)

> an on-link router already advertises an MTU in an RA. I'm not sure if
> it would make sense if it advertised per neighbor MTU options, and how
> those should be interpreted.

The MTU advertised in the RA is for the link, not for the router.  In
particular, even when doing higher first-hop MTUs, this is the value for
all multicast packets.

If the router advertises a higher value through a separate new option,
the failure modes of that are very non-graceful:  a "non-jumbo" ethernet
switch in the middle causes every individual destination's PLMTUD to
fail independently, without the system having any chance to detect it as
an issue towards the router.

-David

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 14 Apr 2016 15:22:11 +0200 (CEST)

> for direct neighbors, not for any random internet host.

I don't believe the draft says this.

What it does is discover the MTU to an IPv6 address on the same LAN. To 
then infer that any packets sent VIA this node, is not something the draft 
(at least explicitly) says. The case could be made to say that any packets 
sent via the "discovered-to-support-larger-MTU-router" could use this 
higher MTU, but that's something that isn't obvious, and I think it's up 
for discussion.

So if I have this network:

H1 - R1 - R2 - Internet

If R1 announces in RA to the LAN where H1 is, that MTU is 1500, but H1 but 
R1 also sends HintMTU of 9180, should the default route H1 installs 
based on this RA and points to R1 also have this higher MTU? Maybe. Maybe 
not.

As I said in the session, I would like more fine granular MTU, so that I 
can have separate MTU for separate entities.

For instance, if I know that most of my devices in my home support 9180, 
but none of my Internet connections do (they only support 1500), then I 
would like H1 to have this in its routing table:

Local /64 HintMTU 9180
RIO with home /56 HintMTU 9180
RIO with ::/0 with HintMTU 1500

Because of this, H1 wouldn't need to spend a lot of time trying to send 
packets to the Internet with larger MTU than 1500.

This would however require that we allow MTUs tied to not only PIO, but 
also RIO. I still don't know what to do about the implicit default route 
that one points to anyone one sees an RA from. Personally I consider this 
design of the IPv6 protocol to be unfortunate.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: David Lamparter <[email-masked]>
Date: Thu, 14 Apr 2016 15:35:48 +0200

> I don't believe the draft says this.

It doesn't, but by tracking neighbors as pairs of (L2 address, IP
version) it behaves pretty much like that.

> based on this RA and points to R1 also have this higher MTU? Maybe. Maybe 

As mentioned in my other mail: The RA's MTU is for the link, not for the
router.  There is no MTU value for the route or router itself;  and
since the RA MTU value doesn't provide any number and we're bumping the
"effective" interface MTU towards the router, that does pretty much
imply the route to R2/internet inherits that value.

> can have separate MTU for separate entities.

Well - host-wise, doing PLMTUD as Ole suggest would probably work out
fine.  Need routers to make the problem interesting ;)

> design of the IPv6 protocol to be unfortunate.

Hrm... indeed an annoying problem...

-David

---

From: [email-masked]
Date: Fri, 15 Apr 2016 10:45:05 +0200

Mikael,

> This would however require that we allow MTUs tied to not only PIO, but also RIO. I still don't know what to do about the implicit default route that one points to anyone one sees an RA from. Personally I consider this design of the IPv6 protocol to be unfortunate.

When you get MTU path discovery in BGP advertisements then we'll put it in the RIO. ;-)

Ole

---

From: David Lamparter <[email-masked]>
Date: Fri, 15 Apr 2016 15:24:03 +0200

> When you get MTU path discovery in BGP advertisements then we'll put it in the RIO. ;-)

BGP is TCP, right?  Let's put PLMTUD on it :3

-David

---

From: [email-masked]
Date: Mon, 18 Apr 2016 09:18:39 +0200

Dave,

> BGP is TCP, right?  Let's put PLMTUD on it :3

You could, but I was more thinking of including MTU information along with prefix advertisements in BGP. Each BGP writer then rewrites the announced MTU if it's interface MTU is lower. That would be equivalent to associating it with an RIO. Unfortunately the BGP signalling path might not be the same as the data path. ;-)

Cheers,
Ole

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 20 Apr 2016 15:35:19 -0700

> what would be gain be to have a specific protocol to probe directly connected neighbors?

Ole,

AFAIR PLMTUD has an upper limit which is the interface MTU.
Thus at least an implementation would need to handle per neighbor MTU 
instead of per-interface. But that doesn't require any new protocol 
behavior.

If we suppose a mixture of nodes on a link - some with per-neighbor and 
some per-interface, then I think things will work slightly more 
efficiently if a per-neighbor node can easily tell whether to bother 
with something larger than the default interface MTU (often 1500); 
receiving some new flag or MTU value in a NS/NA would be sufficient as a 
hint to try larger.
That might also help with corner cases where some old NICs fail/reset in 
interesting ways if they receive a 9k frame.

FWIW I think it is a good idea to couple this with implementing PLMTUD. 
That way at most we need a hint to try larger with the actual probing 
being done as part of PLMTUD.

    Erik