Subject: [Int-area] tunnel MTU and fragmentation

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Aug 2015 16:11:53 +0000

Hi Joe,

For tunnel MTU and fragmentation, we now have two approaches:

https://datatracker.ietf.org/doc/draft-ietf-intarea-tunnels/
https://datatracker.ietf.org/doc/draft-templin-aerolink/

The first approach sets a fixed tunnel ingress MTU based on the egress
reassembly buffer size, and uses fragmentation in case the path MTU
within the tunnel is deficient. It requires that the ingress have some
way of knowing the egress reassembly buffer size, and can result in
a significant amount of fragmentation. Also, it may not match well
with the actual path MTU within the tunnel.

The second approach sets the tunnel ingress MTU to the largest IP
packet size (minus encapsulation overhead) and leverages standard
path MTU discovery whenever possible. Otherwise, it uses limited
fragmentation to ensure that packets up to 1500 bytes get through,
and admits larger without fragmentation in case they are able to
traverse the tunnel in one piece. It only requires the egress to
reassembly at most 1500 bytes. However, host applications that
bind directly to such a tunnel would need to be careful about how
they set the initial TCP MSS.

Should we discuss this, or just let it be?

Thanks - Fred
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Aug 2015 10:41:25 -0700

Hi, Fred,

On 8/12/2015 9:11 AM, Templin, Fred L wrote:
> way of knowing the egress reassembly buffer size,

That knowledge exists; it is based on the requirements of IPv4 and IPv6.

> and can result in
> a significant amount of fragmentation. 

It *can* as a backup plan, but it allows for optimizations based on PLMTUD.

> Also, it may not match well

That's already commonly true for many link layers. Again, however, when
such matching is possible with optimizations, that's permitted by that
draft.

> path MTU discovery whenever possible.

intarea-tunnels sets the tunnel ingress MTU to the egress reassembly MTU
minus encapsulation.

I'm not sure I understand the term "largest IP packet size" - do you
mean "largest unfragmented L2 payload"? - if so, that's not tenable for
IPv6 over IPv6. It would require that the tunnel ingresse IPv6 MTU be
1280-40, but that would be an MTU that would no longer support the
requirements of IPv6.

Both docs, AFAICT, permit MTU discovery to be used as an optimization.

> Otherwise, it uses limited

For IPv6, a guarantee larger than 1280 is not necessary.

> traverse the tunnel in one piece.

Both docs allow this when discovery deems it possible.

> It only requires the egress to
> reassembly at most 1500 bytes.

intarea-tunnels relies on the existing IPv6 requirement for reassembly
(1500 bytes) and the existing IPv4 requirement for reassembly (576).

Requiring IPv4 egresses to reassemble 1500 byte packets would require an
update to RFC791 (and probably RFC1122). Such updates are out of scope
for intarea-tunnels.

> However, host applications that

That's always true in all cases, AFAICT.

> Should we discuss this, or just let it be?

We can certainly discuss it. There's no reason that the intarea-tunnels
needs to constrain aerolink if aerolink also updates RFCs 791 and/or
2460 as needed. intarea-tunnels is not scoped to include such updates.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Aug 2015 19:27:45 +0000

Hi Joe,

I will respond in more detail, but right from the start I do not agree
that what is being proposed in AERO represents an update to RFCs
791, 1122 or 2460. AERO talks about the reassembly buffer required
by tunnel egress' that comply with the recommendation - that is not
the same thing as the minimum IP reassembly buffer sizes, which
are still governed by the base IP RFCs.

Thanks - Fred

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Wednesday, August 12, 2015 10:41 AM
> To: Templin, Fred L; [email-masked]
> Cc: [email-masked]
> Subject: Re: tunnel MTU and fragmentation
> 
> Hi, Fred,
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Aug 2015 13:05:53 -0700

> are still governed by the base IP RFCs.

So is AERO reassembly governed by IP reassembly or a shim layer (ala
SEAL)? If the latter, I agree it wouldn't involve a change. If the
former, you'd be stating that the requirements for IP reassembly for a
tunnel egress would exceed that of the base IP RFCs.

I see no good reason to inject that requirement when it can be avoided.
IMO, tunnel egresses ought to need only support existing Internet host
and gateway requirements.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Aug 2015 21:24:52 +0000

Hi Joe,

> Sent: Wednesday, August 12, 2015 1:06 PM

Either way, shim layer or IP layer, AERO is only telling about what is
required of nodes that configure a tunnel egress. The base IP specs
only specify the minimum reassembly buffer size, and nodes are
permitted to reassemble more. AERO does not need to update
the base IP specs when it suggests a larger reassembly buffer
size for nodes that configure a tunnel egress.

> and gateway requirements.

For IPv6, that would be 1500 bytes at a minimum. And, 576 bytes
for IPv4. That means that a minimum implementation would be
able to reassemble only (1500-ENCAPS) / (576-ENCAPS) per
the IP protocol version specification. But now, if the original
source is located in some other administrative domain than the
tunnel ingress (e.g., somewhere out in the Internet), it might
not receive PTB messages if it sends a 1500 byte packet toward
the final destination via the tunnel.

That is why I am saying that, if the original source is located in
a different administrative domain than the tunnel ingress, the
tunnel ingress should ensure a minimum 1500 even if a little
fragmentation is necessary.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Aug 2015 14:56:15 -0700

Jumping down to the core:

On 8/12/2015 2:24 PM, Templin, Fred L wrote:
...
> the IP protocol version specification. 

That's consistent with intarea-tunnels and with existing IPv4 and IPv6
specs then.

...
> fragmentation is necessary.

In that case, would the key difference be that AERO expects a reassembly
min of 1500 rather than 1460 or 556?

If that's true, then that would be the difference:

	intarea-tunnels describes the requirements for tunneling
	based only on existing host and router requirements

	AERO describes a preferred way of tunneling when
	larger egress MTUs can be expected

In that case, AERO isn't inconsistent with intarea-tunnels unless it
*requires* the larger egress MTUs. In that case, it wouldn't update
IPv4/IPv6 as much as exclude IPv4/IPv6-compliant hosts as egresses.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Aug 2015 22:04:53 +0000

Hi Joe,

> Sent: Wednesday, August 12, 2015 2:56 PM

More properly, a reassembly minimum of (1500+ENCAPS) rather than
a reassembly minimum of 1500 or 576.

> IPv4/IPv6 as much as exclude IPv4/IPv6-compliant hosts as egresses.

There you go again. AERO is not updating the base IP specs when
it says that a node that configures a tunnel egress needs to be
able to reassemble at least (1500+ENCAPS). That is in no way
excluding IPv4/IPv6 compliant hosts.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Aug 2015 15:29:07 -0700

> a reassembly minimum of 1500 or 576.

OK, so it is *larger* than the current expected minimums.

> excluding IPv4/IPv6 compliant hosts.

A host is IPv4/IPv6 compliant if it can reassemble 576/1500 byte
payloads. For IPv6, you're requiring the egress to support 1500+ENCAPS,
which is larger than 1500.

I agree that doesn't mean you're updating IPv4/IPv6 specs (note that I
didn't say that in this response), but it very clearly means that a
compliant host (i.e., with a reassembly max of 1500) would not be able
to support AERO.

That is an exclusion of at least *some* IPv6 hosts. intarea-tunnels
makes no such exclusion, and I see no reason to do so.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 12 Aug 2015 22:48:31 +0000

Hi Joe,

> Sent: Wednesday, August 12, 2015 3:29 PM

It is not too much of a stretch to say that one would only configure
a tunnel egress on a node that is capable of supporting a tunnel.
In this case, a node that is capable of reassembling at least
(1500+ENCAPS).

That said, if the original source, tunnel ingress and tunnel egress are
all within the same well managed administrative domain, then the
egress does not need to be capable of reassembling more than 1500
because the ingress will be permitted to send PTB with MTU
(1500-ENCAPS) or smaller to the original source with reasonable
expectation that the PTBs will be delivered and that the original
source will heed them.

> makes no such exclusion, and I see no reason to do so.

Again, to avoid black holing, when the original source is located in
some other administrative domain (e.g., somewhere out in the
Internet) the tunnel must be made capable of passing all packets
up to 1500 bytes even if a little fragmentation is necessary.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Aug 2015 16:06:34 -0700

On 8/12/2015 3:48 PM, Templin, Fred L wrote:
...
> In this case, a node that is capable of reassembling at least

intarea-tunnels describes how *any* IPv4 or IPv6-compliant host can
support tunneling.

...
> up to 1500 bytes even if a little fragmentation is necessary.

intarea-tunnels trades the goal of efficiency (i.e., avoiding
fragmentation) for the benefit of ubiquity (all IPv4 or IPv6 hosts can
participate as ingress or egress).

That ubiquity is, IMO, the basis of IP.

I appreciate that, in some contexts, there are other trade-offs that
might be useful. That's already noted in intarea-tunnels, though, so I
don't see these two docs as being fundamentally inconsistent.

intarea-tunnels focuses on support for tunnels relying only on existing
requirements. AERO explains the conditions under which a more efficient
solution can be achieved.

Is there an inconsistency you see in that?

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 13 Aug 2015 20:57:54 +0000

Hi Joe,

> Sent: Wednesday, August 12, 2015 4:07 PM

But, it is allowing the tunnel ingress router/host to send PTBs that report
a size smaller than 1500. When the original source is not within the same
well-managed administrative domain as the tunnel, these messages may
be lost and a black hole occurs. This means that the tunnel messes up the
original source's reasonable expectation that packets no larger than 1500
bytes will make it to the final destination or a suitable PTB message will
be returned.

In this case, it is the tunnel's fault that the original source's reasonable
expectations are not upheld. So, the tunnel needs to make sure that
1500 and smaller always get through even if fragmentation is needed.

> solution can be achieved.

I have read your draft, and I think it would be reasonable to ask you to
read AERO; everything you need to see is in Section 3.13.

> Is there an inconsistency you see in that?

One thing that is inconsistent is that I believe we have established that
MTU probing per [RFC4821] cannot be applied for tunnels, since there
is no way to ensure that the probe packets take the same path as
the data packets. The tunnel is the source of the delivery packets;
it is not the source of the transit packets. So, if the network acts on
flow and/or traffic class information within the transit packets the
data packets may take a different path than the probe packets.

Another inconsistency is that AERO recommends the use of
standard path MTU discovery when the original source, ingress
and egress are all within the same well-managed administrative
domain. Standard path MTU discovery is always best when the
source can rely on accurate and assured PTB message delivery.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Aug 2015 14:27:26 -0700

On 8/13/2015 1:57 PM, Templin, Fred L wrote:
...
> a size smaller than 1500. 

Yes, but never a size smaller than 1280 (for IPv6) or 576 (for IPv4).
That's consistent with how all physical interfaces can behave.

> When the original source is not within the same
> be lost and a black hole occurs.

That's consistent with how all physical interfaces can behave.

> This means that the tunnel messes up the

The source should never have had a "reasonable expectation" of 1500
bytes. For IPv6 the expectation is 1280 and for IPv4 the expectation is
576. Everything else requires verification.

> expectations are not upheld. 

The source exceeded the expectations set forth in RFCs 791, 1122, and
2460. The tunnel is no more at fault than would be a physical interface.

> So, the tunnel needs to make sure that

That's true only for 1280 / 576. It has never been true for 1500 unless
the IPv6 source-fragmented, at which point the fragments would get
through as long as none were larger than 1280.

> read AERO

There have been 62 revs of that doc. Do you expect the WG to track that?
Further, IMO this discussion is about the WG intarea-tunnels doc. If I
am mistaken, we don't need to continue.

> ; everything you need to see is in Section 3.13.

I reviewed that in version 62 just now. I see no reason that there
should be a difference between tunnels that cross or do not cross an
administrative domain; that distinction is not a guarantee about ICMP
traversal.

Further, that section introduces many requirements on encapsulation that
I do not agree are necessary for a valid IPv4 or IPv6 tunnel.
intarea-tunnels is intended to allow a variety of solutions, not to
focus on any one particular approach.

> the data packets. 

But that's then true for any PLMTUD. It would never determine a path
MTU, only an MTU for a specific flow.

Again, tunnels don't create a new problem here.

> The tunnel is the source of the delivery packets;
> it is not the source of the transit packets.

These are not terms I use, and I don't see them defined in your doc.

Intarea-tunnels defines "tunnel link packets" as those sourced by the
tunnel (after encapsulation) and "tunnel transit packets" as those
sourced by the origin of the traffic.

> So, if the network acts on

If the network acts on information inside the tunnel packet rather than
the tunnel header, it deserves what it gets. The flow is from the tunnel
ingress to the tunnel egress and is either a distinct flow or an
aggregate of flows.

If it is a distinct flow, then each flow might benefit from separate
PLMTUD. But again that's no different from how any other network
interface operates.

> domain. 

Other than "where ICMP PTBs work", there is no objective definition of
"well-managed administrative domain", so IMO that's not a useful
protocol distinction.

> Standard path MTU discovery is always best when the

You can't ever know when path MTU discovery is working but silent vs.
ICMP PTBs are dropped. Because of this, it should never be assumed as a
solution. Yes, when you know it can work it can be used, but that's
consistent with intarea-tunnels.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 13 Aug 2015 22:04:52 +0000

Hi Joe,

> Sent: Thursday, August 13, 2015 2:27 PM

Right, but physical interfaces that configure an MTU smaller than 1500
bytes and accept packets from nodes out in the wider Internet are
rare. That is why 1500 has become just as real of a number as 1280 and
576 in practice. Tunnel interfaces are being seen as the primary culprits
for path MTU black holing, and they can do something about it. Namely,
pass all packets up to 1500 bytes in length 

> 576. Everything else requires verification.

1500 has become the de facto Internet cell size. We can either ignore
that and continue to black hole, or face up to it and do something about it.

> 2460. The tunnel is no more at fault than would be a physical interface.

Just as sources ignore the IP ID uniqueness principles, they have also
become conditioned to having 1500 byte path MTU or a suitable PTB
message returned.

> through as long as none were larger than 1280.

See above.

> am mistaken, we don't need to continue.

No, this should be about both documents.  Personally, I believe
what is in AERO now is what should be in intarea-tunnels.

> administrative domain; that distinction is not a guarantee about ICMP

Like I said, standard path MTU discovery is always best *when you
can count on it* because the data packets themselves are "probes"
that will eventually help to converge to the minimum path MTU
within the tunnel no matter what ECMP / LAG multipathing may
be present. When you can't count on standard path MTU discovery,
you are obliged to perform fragmentation.

> focus on any one particular approach.

I don't know what you mean here.

> MTU, only an MTU for a specific flow.

No, it is not true for any PLMTUD. When it is the original source that is
doing PLMTUD, the data packets *are* the probe packets, so the
probes and data are guaranteed to take the same paths. When the
tunnel tries to do PLMTUD, the probe packets and data packets
are different, and they might take different paths.

> Again, tunnels don't create a new problem here.

As I said above, tunnels are different because the probe packets
are different than the data packets and can take different paths.

> These are not terms I use, and I don't see them defined in your doc.

I got the terms from the GRE drafts. I think we need a better and
agreed on terminology. "Tunnel link packets" does not sound right
to me.

> ingress to the tunnel egress and is either a distinct flow or an

Yes, but RFC 2983 and 6438 tell how the information within the transit
packet can color the flow information in the delivery packet. And, it
has been suggested to me that even if 2983/6438 are not applied
some networking gear might deep packet inspect and act on flow
information in the delivery packet. getting what it deserves in this
case does not apply to the network because it is the original source
that suffers and not the network.

> PLMTUD. But again that's no different from how any other network

It is different because, as you said, the tunnel aggregates flows.
So the characteristics of each individual flow cannot be subdivided.

> "well-managed administrative domain", so IMO that's not a useful

You are right; a definition of "well-managed administrative domain"
is probably in order:

  well-managed administrative domain - a network where ICMP messages
    are delivered to original sources without possibility of loss due to
    filtering, and where IP source address spoofing is prevented through
    proper application of ingress filtering

> consistent with intarea-tunnels.

That's what I saying; if there is some way of knowing that standard PMTUD
is trustworthy and operational, use it.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Aug 2015 15:38:55 -0700

On 8/13/2015 3:04 PM, Templin, Fred L wrote:
...
...
>> ...but never a size smaller than 1280 (for IPv6) or 576 (for IPv4).
> rare.

But not required.

As a result, intarea-tunnels sets the requirements for tunnels based on
the requirements for IP interfaces.

> That is why 1500 has become just as real of a number as 1280 and

When they do they take a risk based on a capability that exceeds the
requirements. intarea-tunnels doesn't focus on taking that risk but it's
fine if AERO does.

...
> that and continue to black hole, or face up to it and do something about it.

You can make that proposal. Intarea-tunnels is intended to work within
existing requirements only, so such recommendations are out of scope for
that doc AFAICT.

> become conditioned to having 1500 byte path MTU or a suitable PTB

This is the same sort of issue. If you want to design a protocol based
on new assumptions, so be it, but that's out of scope for intarea-tunnels.

...
> No, this should be about both documents.

Intarea-tunnels is a WG doc. AERO is not.

Intarea-tunnels was scoped to assume existing requirements. AERO is free
to make recommendations based on requirements that exceed those
currently required.

That said, I am participating in this discussion to determine issues
with intarea-tunnels.

> Personally, I believe

As noted above, AERO exceeds existing Internet requirements. I don't
agree that such additions are in-scope for intarea-tunnels. I'd like to
hear from others on the list as to their position.

...
> are different, 

Why? They need not be any more different than for any other PLMTUD
algorithm.

> agreed on terminology. "Tunnel link packets" does not sound right

We haven't converged on the terms in intarea-tunnels but we also haven't
really discussed them. Until we do, discussions of the content of that
doc should use those terms.

We should start a separate thread on terminology if you have a suggested
alternate.

> that suffers and not the network.

And routers can do that with packets from end systems, e.g., they can
send DNS A queries in one direction and CNAME queries in a different
one. Short of using encryption, we can't prevent that sort of thing.

> So the characteristics of each individual flow cannot be subdivided.

Any packetization layer that aggregates flows has this issue - e.g.,
using one TCP connection for multiple transfers with different
properties (web transfers and audio), etc.

Section 10.3 of RFC 4821 presents this issue, e.g., explaining how to
deploy PLMTUD for packet fragmentation. Tunneling is no different.
(I will add a ref to that discussion in the next rev).

>     proper application of ingress filtering

Sure, that's tautological but cannot be objectively established without
omniscient access.

I.e., how do you know when you are within the same WM-AD? If you see
PTBs, you are, but if you don't you can't know.

In that regard, WM-AD is about as useful as just saying "use PMTUD".
Both fail silently.

> is trustworthy and operational, use it.

Yes, but how can you *know*?

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 13 Aug 2015 23:07:07 +0000

Hi Joe,

> Sent: Thursday, August 13, 2015 3:39 PM

I will give you one example of an IP link that would configure an MTU
smaller than 1500 - a VHF aviation data link used for controller/pilot
flight critical coordination. Bandwidth is something on the order of
32Kbps. A good candidate for a minimum 1280 MTU. I am not aware
of any operational deployments of IPv6 over VHF aviation data links,
but I can guarantee you if there were they would *not* be talking
directly to correspondents out on the public IPv6 Internet.

> requirements. intarea-tunnels doesn't focus on taking that risk but it's

We need to establish a "threshold" for when it is OK for larger packets
to be dropped but PTB messages go missing, while everything smaller
gets delivered. IMHO, that threshold should be 1500 bytes.

The recommendation then is simply: "Hosts that send packets larger
than 1500 bytes are RECOMMENDED to employ PLMTUD per RFC4821".

> existing requirements only, so such recommendations are out of scope for

Intarea-tunnels is what we make of it, and declaring anything out of scope
right now would be premature. You were quick to point out that AERO has
gone through a number of revisions, but as near as I can tell Intarea-tunnels
has been dormant for years. Our understanding of tunnel MTU and
fragmentation has evolved considerably over that time.

> hear from others on the list as to their position.

My position is that 1500 is a very real number that needs to be
honored in the same way as 1280 and 576. 1500 and smaller
need to either get through or a suitable PTB message returned.
For 1501+, the original source is RECOMMENDED to do RFC4821.

> Why? They need not be any more different than for any other PLMTUD

What do you mean "why"? That's just the way it works [RFC4821].

> (I will add a ref to that discussion in the next rev).

Tunneling is different because the tunnel aggregates potentially
many flows. Probes between tunnel endpoints therefore have
no way of representing all potential flows.

AFAICT, PLMTUD and tunnels do not mix.

> Sure, that's tautological but cannot be objectively established without

This is an operational consideration, i.e., the same as for operational
considerations in setting the MTUs of links in a well-managed network.
If there is operational assurance that PMTUD works, then use it.

> Yes, but how can you *know*?

Operational assurance. If you have operational assurance, use standard
PMTUD. If you don't - don't. This is consistent with clause 3.3 of RFC4459.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Aug 2015 16:52:39 -0700

On 8/13/2015 4:07 PM, Templin, Fred L wrote:
...
> directly to correspondents out on the public IPv6 Internet.

This is irrelevant.

There is simply no requirement that IPv6 links support 1500 byte MTUs.
Until there is, it would be a burden to assume that tunnels also support
those MTUs.

Note that whether you "assume" 1500 or use the current requirement of
1280, if tunnel transit packets have the same MTU as tunnel link packets
the requirement for fragmentation and reassembly - avoided only if
PLMTUD works or PMTUD can be assumed to work - remains.

However, if you assume 1500 then you cannot use an encapsulation shim
layer; you would need to split the tunnel transit packet into the tunnel
link packets and reassemble ONLY the tunnel transit packet at the egress.

Again, there's nothing bad about this except that it exceeds existing
IPv6 requirements and that's out of scope for intarea-tunnels.

> gets delivered. IMHO, that threshold should be 1500 bytes.

Based on what *existing* IPv6 requirement?

> than 1500 bytes are RECOMMENDED to employ PLMTUD per RFC4821".

I would support that recommendation for 1280, but again that has nothing
uniquely to do with intarea-tunnels.

> right now would be premature. 

The scope was determined when the doc was accepted as a WG doc.

> You were quick to point out that AERO has
> has been dormant for years. 

Due to lack of feedback and activity by the WG.

> Our understanding of tunnel MTU and

Yes, the community as a whole is more ready to participate in the
discussion now.

Regardless, though, the scope of intarea-tunnels is to document how to
tunnel IP packets based on existing standards.

...
> For 1501+, the original source is RECOMMENDED to do RFC4821.

Feel free to write that and see if the WG is interested.

> What do you mean "why"? That's just the way it works [RFC4821].

See Section 10.3 of RFC 4821. IP fragmentation issues deal with exactly
the same kind of aggregation as tunnels and the same flow issues. This
is thus already covered in that RFC.

...
> many flows.

How is that different from a network interface?

> AFAICT, PLMTUD and tunnels do not mix.

Again, Section 10.3 of RFC 4821 indicates otherwise.

...
> If there is operational assurance that PMTUD works, then use it.

Sure. And I agree that should be noted in intarea-tunnels as well; I'll
put some text on both this and sec 10.3 of RFC 4821 in the next rev.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 13 Aug 2015 17:07:01 -0700

One other issue we haven't really discussed directly yet:

	it is not feasible to support tunneling unless
	the egress can reassemble a packet that is larger than
	the ingress MUST accept

I.e., the existing IPv6 requirements are OK because the largest tunnel
transit packet that the ingress MUST receive is 1280 bytes and the
egress MUST reassemble up to 1500 byte tunnel link packets. That leaves
220 bytes for the tunnel encapsulation header - IPv6 headers, header
extensions, and shim layers.

We cannot assume that the ingress will transit 1500 byte tunnel transit
packets unless we ALSO assume that the egress can reassemble packets
that are larger.

Although AERO addresses this in Sec 3.13.2, that too is higher than
current IPv6 requirements.

So picking 1500 bytes is just "rearranging the deck chairs*"; it doesn't
solve the fundamental issue that when the traffic source uses the
"largest" packet then tunnels will end up needing to support
fragmentation and reassembly until either PLMTUD or PMTUD can safely
establish otherwise.

So there's no real gain, IMO.

Joe

*for the benefit of those not familiar with the euphemism:
https://en.wiktionary.org/wiki/rearrange_the_deck_chairs_on_the_Titanic

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 15:47:05 +0000

Hi Joe,

> Sent: Thursday, August 13, 2015 4:53 PM

Not it's not irrelevant. Physical links that configure an MTU smaller than
1500 are very likely to be performance-constrained such as the example
I gave. And, those links are very likely to be cordoned off from the public
Internet to avoid DoS exposure. For those links, then standard PMTUD
can be assumed because nodes that are allowed to talk to those links
can be assumed to be within the same well-managed administrative
domain. 

> Until there is, it would be a burden to assume that tunnels also support

Links supporting 1500 MTUs is common operational practice that, at
least IPv6-in-IPv4 tunneling relies on. Even in RFC2460, it says:

   "Links that have a configurable MTU (for example, PPP links [RFC-
   1661]) must be configured to have an MTU of at least 1280 octets; it
   is recommended that they be configured with an MTU of 1500 octets or
   greater, to accommodate possible encapsulations (i.e., tunneling)
   without incurring IPv6-layer fragmentation."

> PLMTUD works or PMTUD can be assumed to work - remains.

This was never in question, and irrelevant for this discussion.

> link packets and reassemble ONLY the tunnel transit packet at the egress.

This has morphed into a different line of discussion that what we were
talking about previously. Of course, if there is a 1500 link and you are
trying to fit a 1500 tunnel over it you will need to fragment. But, that
is a different discussion and not relevant to what we are talking about
htre.

> Based on what *existing* IPv6 requirement?

Based on operational expectations of the vast majority of hosts in the
Internet today. The expectation is that 1500 byte packets will get through
or a suitable PTB message returned. If neither of these conditions can be
upheld, the operational expectations are violated.

> uniquely to do with intarea-tunnels.

See above. Stopping at 1280 allows for the possibility that 1281+
packets don't make it through *and* no PTB message is returned.
Instant black-hole. 

> The scope was determined when the doc was accepted as a WG doc.

There has been a lot of history made since that occurred; mostly by me.
I think you have probably been watching from the sidelines and learning
from my mistakes and making adjustments. I have been the man in the
arena - you have been the critic. 

> Due to lack of feedback and activity by the WG.

There has been no lack of activity on my part. But, what I have been
doing has been aside from the dormant intarea-tunnels and an open
area of investigation that I think you have been following.

> Feel free to write that and see if the WG is interested.

There was at least one earlier indication that the community could
warm to this recommendation:

https://mailarchive.ietf.org/arch/msg/int-area/doCGVsEAEg7oW_dVoq8BenxzABM

> is thus already covered in that RFC.

No, this is different. Section 10.3 deals with adjunct probe packets sent
by the same source that originates the data packets.  With tunnels, the
ingress would have to  employ adjunct probing but it has absolutely no
control over the source(s) of data packets. It is therefore very natural
that the data packets could take different paths than the probe packets,
and the tunnel ingress has absolutely no control over the situation.

> Again, Section 10.3 of RFC 4821 indicates otherwise.

It is different. See above.

> Sure. And I agree that should be noted in intarea-tunnels as well;

I am not commenting on intarea-tunnels. This discussion (started by me)
is about tunnel MTU and fragmentation - not about any one specific draft.

> I'll put some text on both this and sec 10.3 of RFC 4821 in the next rev.

Section 10.3 of RFC4821 does not apply to tunnels. The tunnel ingress has
absolutely no control over the source(s) of data packets, so its adjunct
probing mechanism has no way of determining the paths that data packets
will take. (This is only true for tunnel ingress' that do not occur on the same
node that produces the data packets, but that is now and will continue to
be a widely deployed scenario.)

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 15:52:00 +0000

Hi Joe,

> Sent: Thursday, August 13, 2015 5:07 PM

This is true in the same way that (1280+ENCAPS) is higher than current
IPv4 requirements when tunneling IPv6 over IPv4. And, there are many,
many published standards-track documents that depend on this.

> fragmentation and reassembly until either PLMTUD or PMTUD can safely

1500 is the current internet cell size. We need to honor that, while still
providing mechanisms that allow us to go bigger. Again, there are 3
"magic numbers" in the Internet today: 576, 1280 and 1500.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 14 Aug 2015 09:39:39 -0700

On 8/14/2015 8:52 AM, Templin, Fred L wrote:
...
> many published standards-track documents that depend on this.

When tunneling IPv6 over IPv4, the expectations of IPv4 transit exceed
current standards. There's no way to avoid that, but it should have been
clearly stated as an addition to IPv4 requirements in those
standards-track docs.

That's not needed for tunnels so it can be avoided.

> "magic numbers" in the Internet today: 576, 1280 and 1500.

You can make that case in a separate document. intarea-tunnels is
informational and might be at best a BCP; it cannot override existing
standards.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 14 Aug 2015 09:53:11 -0700

On 8/14/2015 8:47 AM, Templin, Fred L wrote:
...
> Internet to avoid DoS exposure.

This is anecdotal at best.

1500 is the typical size of Internet packets except where it isn't, and
it isn't *all over the place*. Notably over any tunnel, when IPsec is
used, etc. Those are all over Internet.

> For those links, then standard PMTUD

Again, anecdotal. There's no such thing as a "well-managed AD"; your
definition is tautological:
	a WM-AD is one where ICMPs aren't blocked,
	so ICMP can be assumed throughout a WM-AD

The real issue is that there's no objective way to find the boundaries
of a "WM-AD", so unless YOU manage it, YOU cannot assume it exists.

>    without incurring IPv6-layer fragmentation."

Recommended isn't required. We should not be writing descriptions of
required capabilities based on recommended configurations.

> is a different discussion and not relevant to what we are talking about

I agree there's utility in exceeding the minimums and thus that
intarea-tunnels should include those recommendations, but the baseline
requirements should never be based on optional features.

...
> Internet today.

"Vast majority" isn't how we should be designing *minimum* requirements.

> packets don't make it through *and* no PTB message is returned.

Stopping at 1500 has the same exact problem, except that it is based on
"vast majority" rather than requirements.

So there's absolutely no utility in indicating 1500 as a requirement,
and this doc can't indicate new requirements anyway.

...
> by the same source that originates the data packets. 

Section 10.3 deals with IP fragmentation which includes many transport
sources that could have different paths.

At this point, I'd like to hear from others on the list.

Joe

---

From: "Joe Abley" <[email-masked]>
Date: Fri, 14 Aug 2015 13:34:00 -0400

> used, etc. Those are all over Internet.

... particularly at the subscriber edge. PPPoE abounds in many markets. 
Subscriber self-help forums abound with advice on MSS clamping, etc, and 
as any residential helpdesk tech will tell you, you can't rely on PMTUd 
to resolve your sub-1500-byte MTU problems in a world where enterprise 
network people truly and fervently believe all sub-types of ICMP to be 
dangerous and harmful.

[Microsoft Azure doesn't accept or deliver ICMP from/to the Internet, 
for example, and I've been told it's not a bug, it's a feature. To me, 
this is the same as "has no access to the Internet", but I presume to 
the targeted customer base it's "I am protected from harm".]

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 19:34:22 +0000

Hi Joe,

> Sent: Friday, August 14, 2015 9:40 AM

Should have been but wasn't, and still isn't. Many standards-track specs
were published without citing the IPv4 576 minimum reassembly size
and I am sure others are in the pipeline as we speak. That precedent
is already set.

> informational and might be at best a BCP; it cannot override existing

This isn't about overriding existing standards; it is about acknowledging
the current state of affairs so we will have a chance of moving beyond
them. It is long overdue that tunnels begin moving to MTUs greater
than 1500, but to do that we need to shore up the 1500s first. And,
getting to MTUs  larger than 1500 using fragmentation is not what
I am talking about - that would produce too much fragmentation
resulting in too much reassembly.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 19:56:25 +0000

Hi Joe,

> Sent: Friday, August 14, 2015 9:53 AM

Maybe you would find it more acceptable if I said "when there is operational
assurance that standard PMTUD will work", because that is the same sort
of language that occurs in Section 3.3 of RFC4459. I will give you one
example of a domain where there is such operational assurance, and
that is within the Boeing corporate enterprise network where I am
composing this note.

> required capabilities based on recommended configurations.

Not required, but expected - and widely so.

> sources that could have different paths.

Section 10.3 says:

   "However, other more serious failure modes do exist, such as might be
   caused by middle boxes or upper-layer routers that choose different
   paths for different protocol types or sessions.  In such
   environments, adjunct protocols may legitimately experience a
   different Path MTU than the primary protocol.  If the adjunct
   protocol finds a larger MTU than the primary protocol, PLPMTUD may
   select an MTU that is not usable by the primary protocol.  ***Although
   this is a potentially serious problem, this sort of situation is
   likely to be viewed as incorrect by a large number of observers, and
   thus there will be strong motivation to correct it.***"

but there has been at least one indication that middleboxes that
make these sorts of determinations based on looking beyond the
encapsulation headers are already deployed and there does not
appear to be any strong motivation to correct it:

https://mailarchive.ietf.org/arch/msg/int-area/Q-21L9AhheAB10Nwi2sfzI-JkJA

Don't get me wrong - every AERO draft version with the exception of
the most recent has advocated RFC4821 for tunnels. AERO even examined
RFC2983 and RFC6438 implications for not propagating the flow label and
traffic class from the transit packet into the encapsulation headers, but
I am told that even that is not enough.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 20:04:58 +0000

Hi Joe,

> From: Joe Abley [mailto:[email-masked]]
> Sent: Friday, August 14, 2015 10:34 AM
> To: Joe Touch
> Cc: Templin, Fred L; [email-masked]
> Subject: Re: [Int-area] tunnel MTU and fragmentation
> 
> 
> 

What you are saying is that there are situations where hosts that expect
1500 byte packets to get through or a suitable PTB message returned
would not have their expectations met, and so operators have taken
to band-aid patches like MSS clamping. But, if more and more tunnels
support a 1500 minimum then there would be less and less reliance
on band-aids.

> the targeted customer base it's "I am protected from harm".]

The problem with PTBs coming from the Internet is that there is
no way of knowing whether you can trust them. The problem
with PTBs *not* coming from the Internet when they are needed
is black holing. That is why tunnels should do their best to pass
packets up to at least 1500.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 14 Aug 2015 14:00:04 -0700

Fred,

As either informational or BCP, this document cannot make a MUST or
SHOULD recommendation based on "most" or "widely already used"; it needs
to start by stating what we know will always work, based on current
requirements.

This is not the document that will move us forward to a more efficient
future. It is a document that (hopefully) will encourage other tunnel
protocol designers not to do damage by not at least supporting existing
standards.

I'd like to hear what others have to say on these issues. At least one
(Joe Abley) confirmed the issue of the ubiquity of sub-1500 byte MTUs.

Any other viewpoints?

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 21:08:56 +0000

Hi Joe,

> Sent: Friday, August 14, 2015 2:00 PM

But then, I question the need for this document at all. We have had
abundant tunneling specifications for years with many more on the
way. None of those documents have been put on the back burner
waiting for this one to come through. So, what good is it really if
it is not going to make recommendations that stand a chance of
making things better?

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 14 Aug 2015 14:21:28 -0700

> it is not going to make recommendations that stand a chance of

I should expand the abstract to explain this more clearly.

The doc was intended originally as a "one stop shop" to find information
on IP tunneling, to support those creating new tunnel protocols, so they
don't do bad things (e.g., ignoring DF=0 or on arriving IPv4 packets,
creating new tunnel link fragments faster than the ID limits, etc.).

That would make things better, but it was definitely not aimed at
figuring out what we need to *change* in existing protocols to support
tunnels better. I'm not sure I agree there are any such changes that are
useful - as I already explained, any assumption of larger MTUs is
self-defeating and just rearranges the deck chairs to create the same
problem for tunnel link packets, just at the larger MTU.

That's why this doc was originally intended as Informational, and
depending on the guidance it might end up as BCP. However, it does not
specify a specific mechanism and thus can't be standards-track. Because
it's not standards-track, it is constrained by existing standards.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 14 Aug 2015 22:17:56 +0000

Hi Joe,

> Sent: Friday, August 14, 2015 2:21 PM

I see no need for this document if it does not lead to a better Internet.
Tunnel designers have gotten along fine without it for years, and would
ignore it in the future if it doesn't make useful recommendations.
 
> it's not standards-track, it is constrained by existing standards.

But you are recommending the use of RFC4821 probing which would be
a standards track recommendation. And, as I have demonstrated,
RFC4821 probing with tunnels won't work with ECMP/LAG.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 14 Aug 2015 15:48:10 -0700

> ignore it in the future if it doesn't make useful recommendations.

They really haven't.

There have been plenty of tunnels designed that didn't take into account
the need for the egress to reassemble, the need to limit ID reuse, and
the need to not ignore DF=0.

And there are tunnels being designed that exceed current Internet
requirements without being clear about doing so.

Either way, that's the purpose of a baseline.

Again, though, you're one voice. I'd be glad to hear from others.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Aug 2015 15:45:18 +0000

Hi Joe,

> RFC4821 probing with tunnels won't work with ECMP/LAG.

Please respond to this point.  Suppose there are two paths A and B from
the tunnel ingress to egress. Path A has a path MTU of (1500+ENCAPS)
and path B has a path MTU of (1499+ENCAPS). Probes from the ingress
to egress travel over path A, while data packets from ingress to egress
travel over path B. This is a perfectly reasonable multipath configuration,
but the result is a black hole. So, RFC4821 probing is incompatible with
tunnels that are potentially subject to multipath.

Thanks - Fred
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Mon, 17 Aug 2015 11:13:08 -0700

>> RFC4821 probing with tunnels won't work with ECMP/LAG.

There are a number of approaches that might work. The simplest would
involve considering existing packets as probes and using direct
communication to check whether those packets arrive successfully at the
egress.

However, if we consider that such probing will *never* work and don't
have an alternative, we're simply stuck with using the egress reassembly
minimum as specified by the protocol (i.e., this is another reason why
we cannot assume capabilities in excess of those minimums).

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Aug 2015 19:47:01 +0000

Hi Joe,

> Sent: Monday, August 17, 2015 11:13 AM

Using the data packets themselves as probes is the only alternative. But,
to do positive probing for each data packet would be prohibitive in terms
of state maintained by the ingress, network traffic amplification, etc. So,
the only alternative is negative probing - i.e., a message from the network
comes back if a data packet is lost. This is exactly how standard   PMTUD
already works.

The recommendation therefore should be that the ingress admits packets
larger than 1500 bytes with DF=1 and translates any L2 PTB messages that
come back into suitable L3 PTBs to send back to the original source. As we
all know, however, either the L2/L3 PTBs could be lost, and L2 PTBs may
not contain enough information for translation. For these reasons, we
need to say that "Original sources that send packets larger than 1500 bytes
SHOULD use RFC4821 probing to ensure that packets of that size are
reaching the destination".

> we cannot assume capabilities in excess of those minimums).

Stuck with egress reassembly only up to 1500 bytes. There is no reason
to require the egress to reassemble more if the above recommendations
are established.

BTW, "reassembly MTU" in your doc I think is the same thing as
"Effective MTU to Receive (EMTU_R)" as defined in RFC1122. I
think it would be more architecturally consistent to cite RFC1122
and adopt the EMTU_R terminology.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 17 Aug 2015 13:12:02 -0700

On 8/17/2015 12:47 PM, Templin, Fred L wrote:
> to do positive probing for each data packet would be prohibitive

Which is where sampling would be useful.

> larger than 1500 bytes with DF=1 

This would apply only where PLMUTD is implemented.

> and translates any L2 PTB messages that
> come back into suitable L3 PTBs to send back to the original source. 

I disagree; as with PLMTUD, when an error is detected it should affect
subsequent packets. I.e., the tunnel egress MTU would be changed and new
packets that are larger would cause the correct response from the
forwarding layer, with no translation needed.

> As we

Sources sending packets larger than 1280 SHOULD use PLMTUD.

But that is not unique to tunnels, and would be a separate update to
RFC4821.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Aug 2015 21:32:39 +0000

Hi Joe,

> Sent: Monday, August 17, 2015 1:12 PM

You would have to sample every potential flow, and there is no way
of knowing whether a flow you have previously sampled matches
the network's multipath handling for a new packet. In other words,
every packet would have to be a probe. In that case, negative
probing per standard PMTUD is the only alternative.

> This would apply only where PLMUTD is implemented.

As of today, very few hosts are sending packets larger than 1500 bytes to
Internet correspondents. So, we now have an opportunity to say that
hosts that send 1501 or larger packets to Internet correspondents
SHOULD use RFC4821.

> forwarding layer, with no translation needed.

My AERO document currently says exactly the same thing, i.e., cache
the MTU value that the ingress sees in the L2 PTBs (minus ENCAPS)
but also translate the L2 PTBs into L3 PTBs *when possible*.

However, this is an area for further consideration *if* we can achieve
a consensus that hosts that send 1501+ packets SHOULD use PLMTUD.
If the ingress caches the minimum MTU size for all paths to the egress,
then the MTU might be much smaller than that available on some paths.
So, if the ingress does not cache the MTU then some flows may be able
to use, e.g., 8KB packets while others are limited to, e.g., 1500.

> Sources sending packets larger than 1280 SHOULD use PLMTUD.

I don't have a problem with that, but PLMTUD is not widely used today
and we will need to carry along legacy non-PLMTUD hosts for a long
time to come. That means that we need to take care of packets in
the (1280-ENAPS) to 1500 byte range, using fragmentation if necessary.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 17 Aug 2015 14:43:43 -0700

On 8/17/2015 2:32 PM, Templin, Fred L wrote:
> the network's multipath handling for a new packet.

That's always true for PLMTUD, but statistically sampling every 10th or
100th packet might be more than sufficient. If that's not fast enough,
then you shouldn't be using PLMTUD to try to find paths for MTUs larger
than you already know egresses can handle.

I.e., this is an *optional* optimization which has risks.

> In other words,

For negative probing to work, you'd have to know that ICMPs aren't
blocked on any return path, so it's hardly a viable alternative.

> hosts that send 1501 or larger packets to Internet correspondents

I agree in general but this isn't the place for that recommendation.

That would belong in RFC4821-bis or RFC1122-bis.

It isn't specific to tunnels.

> a consensus that hosts that send 1501+ packets SHOULD use PLMTUD.

I hardly see how that helps for tunnels. If you don't run the PLMTUD at
the ingress-egress pair, you can't know whether the ingress can avoid
source fragmentation.

If you do run that at the ingress as a host, then - as you already note
- you have at best a statistical guarantee. That's a reasonable
recommendation, but not as strong as saying that "all hosts" should be
doing PLMTUD.

> to use, e.g., 8KB packets while others are limited to, e.g., 1500.

Sure, but again that is specific to running PLMTUD between the ingress
and egress only.

> the (1280-ENAPS) to 1500 byte range, using fragmentation if necessary.

That's what the doc already recommends.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 17 Aug 2015 22:49:30 +0000

Hi Joe,

> Sent: Monday, August 17, 2015 2:44 PM

That is no good Joe, and I think you know it. There is no way of knowing
whether all paths in a multipath have been sampled before a packet that
takes a degenerate path in the multipath arrives. 

> blocked on any return path, so it's hardly a viable alternative.

That is the reason for the SHOULD use PLMTUD for packets larger than
1500 bytes. We know that hosts currently aren't sending 1501+ packets
to Internet correspondents, so we allow the tunnel to be an unreliable
source of PTB messages for packets in this size range. So, if the tunnel
delivers an ICMP PTB message it is a useful optimization, but not
required since the host is presumably using PLMTUD.

> That would belong in RFC4821-bis or RFC1122-bis.

Why not simply "updates RFC4821/RFC1122/RFC6434"? There is no
need for a (bis).

> It isn't specific to tunnels.

Tunnels would be more susceptible to PTB loss than ordinary links. So,
tunnels would require hosts that send packets larger than 1500 to be
careful about black holes when PTBs are lost.

> the ingress-egress pair, you can't know whether the ingress can avoid

We have already talked about that; running PLMTUD at the ingress-egress
pair is infeasible due to multipath. There may be instances in which the
ingress can know that source fragmentation is avoidable through operational
assurances but, lacking that, source fragmentation for packets in the
[(1281-ENCAPS), 1500] size range is necessary.

> Sure, but again that is specific to running PLMTUD between the ingress

No, it is specific to PMTU assurances for hosts that send large packets.
PLMTUD between the ingress and egress is off the table AFAICT.

> That's what the doc already recommends.

That's what I *want* your document to say, because that is what AERO
currently says. If you agree to the text, then please add it.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Aug 2015 11:44:28 -0700

Fred,

I'm very confused by your position. Let me explain my views as
succinctly as possible below. Can you please indicate briefly where your
views differ and why?

Joe

-----

IMO:

	tunnel requirements should be just that;
	requirements to support tunnels consistent
	with the *existing* Internet architecture
	and requirements

		which means new requirements for
		endsystems that are not tunnel
		endpoints are, IMO, not in-scope

	tunnel ingress and egress MTU is limited
	only by the egress MTU. Focusing on IPv6:
		the pathMTU is 1280
		the egressMTU is 1500
	every other value is an optimization that
	depends on whether the ingress and egress
	have a way to coordinate and *know* that
	a larger MTU works (in either case).

		PLMUTD is one way to know, but
		would be problematic with ECMP
		based on DPI

		PMTUD is another way to know, but
		assumes control over the network
		over which tunneled traffic transits

As a result:

	traffic sources (i.e., of TTPs) have no
	new requirements. yes, if they want to send
	unfragmented packets or fragments larger than 1280
	then they are taking a risk

		yes, they too should use PLMTUD or
		PMTUD, but yes, both can fail in
		exactly the same way as for tunnels

		no, this has nothing to do with traversing
		tunnels

	it would be useful to find a way to avoid fragmenting
	at the tunnel ingress BUT THERE IS NO PROTOCOL THAT
	SUPPORTS THIS

		this would rely on "packet larger than I'd like",
		rather than "packet too big and I *had* to drop it",
		but that signal doesn't exist.

	TTPs between 1280-header..1500-header WILL be source-fragmented
	at the tunnel ingress and reassembled at the egress

		that may be inefficient, but avoiding inefficiency
		is an optimization and that's what further
		ingress-egress coordination would solve

		NO, this should not be solved by new requirements
		on sources. They need to be able to assume
		that 1280-header...1280 packets can transit ALL
		IP-capable links without ANY further action,
		and MUST NOT need to rely on PLMTUD or PMTUD
		for traversal in those cases

		Yes, 1281..1500 byte unfragmented packets may
		not traverse the tunnel at all, but that's a risk
		the source *already* takes for ALL IPv6 links,
		and in those cases, just as in all other links,
		the source ought to be using PMTUD or PLMTUD

---

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Aug 2015 20:42:15 +0000

Joe,

I don't know how to respond to your post on a point by point basis, so I
will try by explaining my views as follows:

1) In environments where standard PMTUD works, use it. That means,
    set the initial ingress MTU to "unlimited" and cache any size limitation
    reported in a PTB messages from a router on the path to the egress. If
    that size is less than (1280+ENCAPS), either commence fragmentation
    or declare the tunnel non-functional.

2) In environments where standard PMTUD might fail, handle packets
    according to their size as follows:

    a) size <= (1280-ENCAPS) - encapsulate and send w/o fragmentation
    b) (1280-ENCAPS) < size <= 1500 - encapsulate and fragment
    c) 1500 < size - encapsulate and send without fragmentation only
                                  if the packet is no larger than the cached MTU

   Clause a) is not surprising because it fits within the IPv6 minimum
   link MTU.

   Clause b) is what tunnels MUST do to get packets up to 1280 through
   and what they SHOULD do to get packets between 1281 and 1500
   through. If they can't fragment for packets 1280 < size <= 1500,
   then they MUST employ  some sort of adaptation such as MSS
   clamping.

   Clause c) requires that original sources that send packets larger
   than 1500 bytes be prepared that the packets may be lost silently
   in the network. That means that sources that send packets larger
   than 1500 bytes SHOULD use RFC4821.

   Note however that some or all of this can be obviated when there
   are operational assurances that all potential links in paths that the
   tunnel may traverse configure sufficiently large MTUs. So, for
   example, if all links are known to configure a 1500 MTU then the
   ingress can send all packets up to (1500-ENCAPS) without
   fragmentation. This is what many tunnels currently do today.

Thanks - Fred

> Sent: Tuesday, August 18, 2015 11:44 AM

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Aug 2015 13:56:41 -0700

Fred,

Working off of your clauses...

On 8/18/2015 1:42 PM, Templin, Fred L wrote:
> 1) In environments where standard PMTUD works, use it.

That's only very specific, controlled environments, so that's a special
case. We agree it's useful.

> 2) In environments where standard PMTUD might fail, 

Basically everywhere else...

>     handle packets

Agreed.

>     b) (1280-ENCAPS) < size <= 1500 - encapsulate and fragment

Agreed.

...
>    in the network. 

See below...

> That means that sources that send packets larger

I agree that any host that sends packets over 1280 really ought to know
it will work - which means they ought to use RFC4821 - that's not a
current requirement.

This seems to have nothing to do with tunnels, though, so I can't see
why this doc should be adding this requirement.

---

That said, getting back to the first point in this clause, IMO a tunnel
ought to handle these packets as a link - either they can transit the
packet by source fragmentation or they cannot.

I.e., IMO if the TTP > egressMTU then the packet MUST be dropped with an
ICMP PTB.

The ingress MUST default to an egressMTU of 1500-encaps. There's no way
to default to anything else.

A larger egressMTU can be determined by explicit coordination or
a-priori knowledge of egress capabilities. If you don't have that
knowledge, you are saying the link can be a blackhole for packets. IMO,
that's what PTB is for - it should drop those packets and send ICMP PTBs
back *until* it knows otherwise.

Additionally, I also don't see how the tunnel can optimize the fragment
size UNLESS it has some knowledge it can rely upon - e.g., controlled
environment or willingness to trust PLMTUD.

I.e., we all know things can get better in controlled environments; it's
the uncontrolled ones we need to handle.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 18 Aug 2015 21:50:25 +0000

Hi Joe,

Good to see there is some agreement. See below for follow-up:

> Sent: Tuesday, August 18, 2015 1:57 PM

OK.

> Basically everywhere else...

Correct.

> >     a) size <= (1280-ENCAPS) - encapsulate and send w/o fragmentation

OK.

> >     b) (1280-ENCAPS) < size <= 1500 - encapsulate and fragment

OK.

> it will work - which means they ought to use RFC4821 - that's not a

Most hosts on the Internet today connect to links with 1500 byte MTUs.
This is the size that they expect will work, or a suitable PTB message
returned. So, the tunnel needs to either uphold this expectation or
perform some sort of adaptation like MSS clamping to mask the
encapsulation overhead.

Most of these hosts probably will not retroactively employ RFC4821
even if we were to publish a BCP recommendation. But, we have an
opportunity to set a recommendation for hosts that connect to 1501+
byte links by saying that they SHOULD use RFC4821 for packets larger
than 1500.

> why this doc should be adding this requirement.

The tunnel recommendations are really the first place we are openly
acknowledging that large packets (1501 or larger) might be silently
lost in the network. The same might be true for other link types,
but for tunnels we absolutely know it is true.

> to default to anything else.

It should default to "unlimited", and then shrink down from there
based on any PTB feedback from a router somewhere within the
tunnel. That is the way RFC2473 works, and the only way to take
advantage of larger MTUs when they are available.

> back *until* it knows otherwise.

Again, knowing the egress' EMTU_R does not allow the ingress to know
the minimum MTU across all paths that contribute to a multipath, so there
is no way to send whole packets up to that size without risk of loss over
one of the paths. And, fragmentation for packets considerably  larger
than 1500 bytes is too much fragmentation - loss of a single fragment
means that the whole packet has to be retransmitted.

When fragmenting packets between 1280 < size <= 1500, we can always
get away with sending just two fragments. For larger size packets, three
or more fragments may be necessary, and that is too many. 

> environment or willingness to trust PLMTUD.

If we place a limit on the largest packet size we will ever fragment
(i.e., 1500 bytes) then we can always be sure that we can send it as
a two-fragment instead of a many-fragment and we don't have to
 worry about optimizing the fragment size. Packets larger than 1500
can go unfragmented or not at all.

> the uncontrolled ones we need to handle.

That is exactly right, yes.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Tue, 18 Aug 2015 15:23:51 -0700

Looks like we're down to clause C:

...
> even if we were to publish a BCP recommendation.

So we agree that:

	- ICMPs are already blocked in many places
	- many hosts do not use PLMTUD

At that point, any host that uses 1500B packets is asking for trouble
and already has to deal with the consequences.

> But, we have an

IMO, hosts sending fragments larger than 1280 ought to use PLMTUD, but
again this has nothing to do with tunnels.

> but for tunnels we absolutely know it is true.

Don't we already know it's true because ICMPs are blocked?

> advantage of larger MTUs when they are available.

It has to default to 1500-encaps unless there are other mechanisms in place.

If there are other mechanisms in place, yes, it can 'default' to other
values (though I wouldn't call that 'default' as much as 'initialize').

RFC2473 is, IMO, in error on many of these points, as is summarized in
Section 5.4.2. I agree it tries to "take advantage" of a larger tunnel
MTU, but this sort of "MTU matching" over L2 is too hazardous unless
there are L2 (tunnel) mechanisms to determine the appropriate values.

I.e., just "assuming you can send larger" isn't a viable strategy unless
you KNOW you can trust PMTUD, PLMTUD, or some OOB protocol between
ingress and egress.

> means that the whole packet has to be retransmitted.

I think you're mixing two issues:

	- the largest packet that can get across a tunnel
		this is governed by the egressMTU

	- the largest TTP that can transit a tunnel
	without source fragmentation of the TLP
		this is an optimization that it's clear cannot
		be assumed; it has to be verified using direct means
		between the ingress and egress

> or more fragments may be necessary, and that is too many. 

I don't understand why you think that the TTP source can - or should -
care or even know about L2 fragmentation efficiency without some
explicit mechanism to do so.

I also disagree that three fragments is too many; the number depends on
what the egress can efficiently handle. As a counterexample, ATM handles
much more than three very easily.

> (i.e., 1500 bytes)

The largest TTP size an ingress MUST fragment is 1280+ENCAPS.

The smallest TTP size an ingress MUST NOT fragment is 1280-ENCAPS.

Given those two values, it's clear that an egress MAY reassemble
two-fragment TTPs up to:

	2 * (1280-ENCAPS) = 2560-2*ENCAPS

Unless ENCAPS is huge, we should be able to assume that 1500B TTPs are
split into no more than two fragments, but I don't see why we'd want or
need to state that.

> then we can always be sure that we can send it as

AFAICT, larger than 1280-ENCAPS go unfragmented only when the
ingress-egress pair knows better.

Otherwise, not at all *and* generate an ICMP PTB.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 16:09:34 +0000

Hi Joe,

> Sent: Tuesday, August 18, 2015 3:24 PM

But, don't most hosts set their TCP MSS based on the MTU of
the underlying interface (so, 1500 minus 20/40 for IPv4/IPv6
header minus 20 for TCP header) ? If 1500 were such a problem,
we would have hosts black holing all over the Internet all the
time, but that does not seem to be the case.

> Don't we already know it's true because ICMPs are blocked?

Tunnels are different because the endpoints might be on the far
side of the Internet from one another, where any hop along the
way could be filtering PTBs.

> It has to default to 1500-encaps unless there are other mechanisms in place.

The mechanism is standard PMTUD. It might fail silently for some
packets over 1500 bytes, but then the original hosts that send
packets larger than that SHOULD use PLMTUD. 

> values (though I wouldn't call that 'default' as much as 'initialize').

RFC2473 says that the default value is the largest possible IP
packet size, minus ENCAPS. That is the same as what I am
suggesting here.

> you KNOW you can trust PMTUD, PLMTUD, or some OOB protocol between

The mechanism is RFC4821 PLMTUD between the original source and
the final destination.

> much more than three very easily.

Here is the crux of the disagreement. It is not just the egress EMTU_R
that determines how large of a packet can traverse the tunnel; it is also
the path characteristics in the path between the ingress and egress.
Consider the "dumbbell" configuration, where the ingress and egress
are heavy weights with a long thin pipe in between. For example, if the
ingress and egress both connect via 10GigE. The ingress has some way
of knowing that the egress has an EMTU_R of 8KB, and so begins sending
fragmented 8KB packets at line rate. But, the network path in between
has a slow link that drops every Nth packet. When a fragment gets
dropped, the performance implications are devastating because the
source host will retransmit the entire 8KB packet again, and not just the
fragment that went missing. And, if the network path loss rate is high
enough then no packets will ever get through and a black hole ensues.

So, the ingress needs some way to learn about the path characteristics
and not just the egress EMTU_R, and the only way is through standard
PMTUD. But, we know that this can fail so the original source has to do
its part by using RFC4821.

> The largest TTP size an ingress MUST fragment is 1280+ENCAPS.

You seem to be going back in yourself. In the last message, we agreed
that the "fragmentation zone" is (1280-ENCAPS) < size < 1500.

> The smallest TTP size an ingress MUST NOT fragment is 1280-ENCAPS.

I would not say MUST NOT to this; "NEED NOT" would be more accurate,
or more properly "MAY send without fragmenting".

> split into no more than two fragments, but I don't see why we'd want or

Two-fragment maximum is indeed what I am saying.

> Otherwise, not at all *and* generate an ICMP PTB.

I will state my rules again just so we can get back on the same page:

  [size <= (1280-ENCAPS)] - Send unfragmented
  [(1280-ENCAPS) < size <= 1500] - Fragment into two fragments
  [size > 1500] - Send unfragmented, or return PTB

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 09:48:09 -0700

On 8/19/2015 9:09 AM, Templin, Fred L wrote:
...
> header minus 20 for TCP header) ? 

For communication within the LAN or as a limit when using PLMTUD/PMTUD.
In the absence of any info, wide-area packets are supposed to be limited
to the IP standard max.

> If 1500 were such a problem,

Right - see above as to why.

> way could be filtering PTBs.

That's true of nearly all Internet paths.

> packets larger than that SHOULD use PLMTUD. 

There's no magic reason to trust PMTUD for packets under 1500 and need
PLMTUD for larger ones. In *both* cases, PMTUD works when it works and
fails silently and PLMTUD works when it works and fails incorrectly in
the presence of ECMP that splits flows that the PLMTUD algorithm treats
as grouped.

> packet size, minus ENCAPS. That is the same as what I am

Yes, and I have explained why I disagree with that recommendation in
both docs.

> The mechanism is RFC4821 PLMTUD between the original source and

It doesn't matter how you get the info, but the info is useless to
govern how the ingress fragments unless the ingress has the information.

The ingress needs to know when 1500 black-holes. That black-holing can
occur all the way down to 1280-ENCAPS. The source should never need to
spin its MTU down below 1280 based on traversal of a tunnel.

> the path characteristics in the path between the ingress and egress.

The egressMTU defines "how large of a packet *can* traverse the tunnel".

Period.

The path characteristics of the ingress-egress (i.e., the tunnel path
MTU) define how the *ingress* needs to fragment packets to get them to
the egress for reassembly.

> enough then no packets will ever get through and a black hole ensues.

We saw this on underprovisioned ATM switches that did tail-drop when the
output ports were overloaded (the solution there was ensure the output
could buffer #inputs * cells-per-packet).

Yes, there are pathological cases like this, and yes, they would be
faulty links that need to be fixed.

> PMTUD.

The ingress can also figure this out by using PLMTUD or by feedback from
the egress that reports that all reassemblies fail.

> But, we know that this can fail so the original source has to do

A source using PLMTUD would get confusing information in this example.
Some packets would get through for any size under N fragments but the
loss could still be VERY high. I.e., let's look at a 2000-byte packet
with 100-byte fragments. If every every 20th fragment is lost, then a
1999-byte packet would get through. The next 2000 (or so) 1999-byte
packets would NOT, then one packet would get through.

So your PLMTUD would optimize for 0.1% efficiency.

If you instead required the ingress-egress to coordinate, you'd have
figured this out and fixed it.

> or more properly "MAY send without fragmenting".

Strictly, it would be "MUST be able to send without fragmenting" is more
accurate?

However, if a source doesn't need to fragment, it MUST NOT (IMO).

> Two-fragment maximum is indeed what I am saying.

What's the value in recommending that?

RFC2460 already requires 1500 as a reassembly MTU. There's nothing in
RFC2460 that requires use of the minimum number of fragments, but that's
clearly a SHOULD for all sources (ingress included).

Changing RFC2460 to say that a max of 2 fragments is recommended is
clearly out of scope here and there's no clear reason to require it for
only ingresses.

----

I'll send a separate post to get us back to the key issue as a group
shortly.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 17:19:44 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 9:48 AM

But they aren't; 1500 packets occur across the wide area all the time.

> the presence of ECMP that splits flows that the PLMTUD algorithm treats

PLMTUD would need to be used on a per-flow basis from the source
host; grouping the flows would not work in the presence of multipath.
Do we need to update RFC4821?

> Yes, and I have explained why I disagree with that recommendation in

This approach is necessary to allow for discovery of path characteristics
within the path from the ingress to the egress. Path characteristics
include the minimum link MTU of all links in the path even if multipath
is in play. This is important, because link MTUs are set according to
performance characteristics - you would never set an 8KB MTU on
a 100Kbps link, for example.

> spin its MTU down below 1280 based on traversal of a tunnel.

The ingress only needs to know that TTPs of (1280-ENCAPS) < size <= 1500
have to be accommodated using fragmentation if necessary. That way,
these sizes will never black hole.

> The egressMTU defines "how large of a packet *can* traverse the tunnel".

The egress EMTU_R is only an indication of the egress' reassembly
capabilities - it says nothing about the MTU-bearing characteristics
of the tunnel path. 

> faulty links that need to be fixed.

No, this is standard operational practice and not at all a pathological
case. There will always be cases where the ingress and egress are
attached to high speed, large MTU links, but the tunnel path in
between has congested links and/or links with small MTUs. Placing
large numbers of fragments in flight over a path where large
numbers of packets are dropped is a recipe for fragment loss
black-holing.

> the egress that reports that all reassemblies fail.

Already told why PLMTUD between ingress and egress is not
workable (multipath). And, feedback signaling from the egress
is just as likely to be lost as PTB messages. 

> So your PLMTUD would optimize for 0.1% efficiency.

The smallest fragment size I intend to allow is 1024 bytes,
for either IPv4 or IPv6. Tunnel source-fragmentation then
produces at most two fragments for each packet.

> figured this out and fixed it.

Ingress/egress signaling is cumbersome and as prone to loss as PTBs.
Consider also when there is one ingress and millions of egress'. How
is the ingress supposed to keep track of timers for all of the millions
of egress'?

> Strictly, it would be "MUST be able to send without fragmenting" is more

That's OK.

> However, if a source doesn't need to fragment, it MUST NOT (IMO).

I would prefer SHOULD NOT.

> What's the value in recommending that?

It means that when the loss probability is non-negligible there is
a much higher probability that both fragments of a two-fragment
packet would be delivered while at least one fragment of a
multi-fragment would be lost.

> clearly out of scope here and there's no clear reason to require it for

Nobody is suggesting changes to RFC2460.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 10:49:03 -0700

Separating the points out:

> Do we need to update RFC4821?

Possibly, but not in this document. Tunnels are no different than IP
fragmentation in the need to potentially measure feedback across
different flows.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 10:51:21 -0700

Separating the threads out:

On 8/19/2015 10:19 AM, Templin, Fred L wrote:
> case. 

It is hardly standard practice to drop every Nth fragment.

A link that cannot transit either a single 1280-byte packet or the
fragments needed to reassemble into a 1500-byte packet is correctly
identified as a black-hole.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 17:57:51 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 10:49 AM
> Subject: tunnel doc - PLMTUD issue
> 

We have already established that, for tunnels, PLMTUD probing won't
work because of multipath, *and* the multipath is not pathological.

Whether you view that as different than IP fragmentation or not
is beside the point that the tunnel has absolutely no way of knowing
whether the probes will follow the same paths as the data packets.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 17:57:51 +0000

Hi Joe,

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 18:05:59 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 10:51 AM
> Subject: tunnel doc - pathological fragment drop cases
> 

Packets are dropped at links when the queue in front of the link
is too full. There are many different drop schemes (RED, tail
drop etc.) but that is beside the point. The point is that if the
packet delivery rate is N%, then (100-N) packets out of every
100 packets will be lost. And, as N decreases the number of
fragments lost increases. And, for every fragment lost the
entire packet needs to be retransmitted, resulting in more
fragments lost, etc.

> identified as a black-hole.

That is why we want to limit to two-fragment packets - fewer
fragments per packet to go missing, and increased chance that
both fragments will arrive.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 11:07:57 -0700

On 8/19/2015 10:57 AM, Templin, Fred L wrote:
> work because of multipath, 

You have made that claim and I have described a way it can still work.

> *and* the multipath is not pathological.

We do agree on that.

> whether the probes will follow the same paths as the data packets.

It does matter - if we can't make it work for IP fragmentation then
PLMTUD needs to be embedded in every application and we can't have
stateless communication anymore.

If we can make it work for IP fragmentation then it works for tunnels.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 11:10:42 -0700

On 8/19/2015 11:05 AM, Templin, Fred L wrote:
...
> is too full. 

That would result in some packets getting through and the link would
work. It wouldn't work well, but E2E PLMTUD won't help at all.

...
> both fragments will arrive.

We don't need to establish a limit if we never need more than that limit
for the default case.

You're suggesting yet another extension to existing host requirements
that "when more than 2 fragments are required" -- do what? drop them and
send PTB? You're then just constraining the link to the size of the two
fragments. Maybe there's good reason to do that everywhere, but this has
no specific relation to ingress packet handling.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 19:15:39 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 11:08 AM
> Subject: Re: tunnel doc - PLMTUD issue
> 
> 
> 

How was that? I must have either missed it or dismissed it.

> stateless communication anymore.

The question is whether sizes discovered through RFC4821 path MTU
probing of a single flow can be aggregated across many flows, and I
don't think that it can because of multipath. But, I don't think that
that is an application layer burden; the transport layer should take
care of it.

> If we can make it work for IP fragmentation then it works for tunnels.

Well, again, RFC4821 says:

   "Although this is a potentially serious problem, this sort of situation is
   likely to be viewed as incorrect by a large number of observers, and
   thus there will be strong motivation to correct it."

but, this statement is incorrect at least inasmuch as ECMP/LAG is 
concerned. The probe packets can take a different path than the
data packets all day long, and an observer would report that they
saw ECMP/LAG working correctly and as designed.

For tunnels, the only reasonable way out then as far as I can tell
is for the ingress to make each data packet also be a probe packet,
and accept negative feedback from the network if a packet is lost
in the form of a PTB message. But, that means that the original
source needs to implement RFC4821 for each flow it produces.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 19:24:46 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 11:11 AM
> Subject: Re: tunnel doc - pathological fragment drop cases
> 
> 
> 

A simple case. Assume a link packet delivery ration of 75%. Means
that 1 out of every 4 packets is lost. Means that we can expect to lose
almost every fragmented packet that includes 4 or more fragments.
But, it means that 50% of two-fragment packets will still get through.
That may not seem like a lot, but consider a case such as high-speed
streaming video. Losing 50% of the frames is always better than losing
100% of the frames.

> no specific relation to ingress packet handling.

The two-fragment applies only to tunnel ingresses that need to squeeze
middling-sized packets across the path to the egress at line rate. Other
applications are certainly welcome to produce isolated many-fragment
packets but would be ill advised to send them at steady state  line rate.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 13:58:57 -0700

On 8/19/2015 12:15 PM, Templin, Fred L wrote:
...
> How was that? I must have either missed it or dismissed it.

Dismissed it.

It was as you note at the end:

	- use existing packets as probes
	- send sampled feedback
	- ingress would need to look for possible ECMP
	effects in the samples

This is identical to what IP fragmentation would need to do at a host.

...
>    thus there will be strong motivation to correct it."

IMO, ECMP using DPI on the inner header of tunnel packets is a serious
error and not the same as persistent ECMP on the outer header.

Inner header DPI ECMP would work only for the first fragment anyway.
It's clearly a bad idea to try to do ECMP on tunneled packets based on
such deep information.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 22:08:26 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 1:59 PM

All packets are probes when you employ standard PMTUD.

> 	- send sampled feedback

You mean the egress ACKs each probe?

> 	- ingress would need to look for possible ECMP

What is meant by "ECMP effects"? A probe either gets through or does not
get through. And, there is no way of knowing whether subsequent data
packets will follow the same path as the samples.

Sampling would also be a major pain in the neck when there is one
ingress and many egress', such as for NBMA tunnels.

In short, even if it works (which I don't think it does) it is much more
cumbersome than required. So yes, I think I dismissed it and still do.

> error and not the same as persistent ECMP on the outer header.

I definitely agree, but someone mentioned that DPI on inner headers
does happen.

> It's clearly a bad idea to try to do ECMP on tunneled packets based on

Again, I agree. But, assuming the ECMP/LAG decisions were only based
on information in the outer header we still have a problem. Namely,
RFC2893 and RFC6438 specify copying flow information from the inner
headers to the outer headers. In order to get probes to follow the same
paths as the data packets we would have to disable the behavior specified
in those RFCs, which would defeat ECMP/LAG altogether - probably not
what the network designers intended.

So, we are up the creek whether or not the ECMP/LAG equipment is
doing DPI.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 15:35:57 -0700

On 8/19/2015 12:24 PM, Templin, Fred L wrote:
...
...
> that 1 out of every 4 packets is lost.

It means that every packet has a 75% chance of getting through. That's
not the same as dropping every 4th packet.

On average, the probability of four adjacent fragments all getting
through would be 0.75^4 = 32%, not 0%. Five fragments would be 24%, six
would be 18%, etc. Yes, things are better for 1 fragment (75%) or two
(56%), but even three gets through 42% of the time too.

The case you want is where exactly every Nth packet is dropped, which is
a conditional probability that rarely happens in a real system. Even
tail drop won't do this unless the incoming traffic has a single packet
size AND somehow packets are taken from the queue in groups of N-1.

Even if such an extreme corner case existed, you end up with very high
losses if you fragment into N-1. In that case, every Nth packet still
gets through. In either case, if *some* packets get through then PLMTUD
won't detect a problem.

When there's per-packet loss it's always better to have fewer fragments.
When the loss is proportional to the packet size, that's not true. And
there are other means of dealing with this at the link layer - e.g.,
FEC. Tunnels, being links, should consider using these methods where
their loss properties warrant.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 15:35:53 -0700

On 8/19/2015 3:08 PM, Templin, Fred L wrote:
...
> All packets are probes when you employ standard PMTUD.

Yes, but PMTUD relies on negative feedback that can get silently
blocked. The benefit of PLMTUD is the need for positive feedback.

> You mean the egress ACKs each probe?

"Sampled" means send confirmation about a subset of those that are
received. The sample size depends on the flow rate.

> What is meant by "ECMP effects"? 

When a set of samples gets through and another set does not and the two
sets vary by a property other than packet length.

> A probe either gets through or does not

That's true, but that's true for all PLMTUD samples from the origin
source too. I.e., that's a property of PLMTUD - it treats success of
past packets as implied success of future packets.

For that matter, the same is true of PMTUD except that the correlation
is negative, i.e., when there's no feedback it assumes that the success
of past packets predicts that future packets will also succeed.

> ingress and many egress', such as for NBMA tunnels.

Sampling is all you can do other than work with the minimums, UNLESS you
*know* otherwise (i.e., you've deployed the network).

> cumbersome than required. 

There's no alternative when you don't know you have an ICMP-preserving
subnet. We agree that when you do, you should use PMTUD.

...
> I definitely agree, but someone mentioned that DPI on inner headers

Bad things happen all over the place. We can mention that it's a bad
idea and has bad effects.

> headers to the outer headers. 

RFC2893 indicates that the IPv4 outer header has a TOS of 0 and
generates the IP source and destination based on the ingress and egress.

RFC6438 indicates how to set the IPv6 flow label to treat each flow as a
separate tunnel; that is a strong argument for running PLMTUD on a
per-flow basis (via sampling).

> In order to get probes to follow the same
> in those RFCs,

You only have to (statistically) track the packets you're already sending.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 23:06:02 +0000

My blunder. Thought I saw something and got off on a wrong tangent.

Thanks - Fred

> Sent: Wednesday, August 19, 2015 3:36 PM

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 19 Aug 2015 23:27:34 +0000

Hi Joe,

> blocked.

Yes; that is why the ingress only uses PMTUD for packets larger
than 1500, and require the original source to use PLMTUD for
packets larger than 1500.

> The benefit of PLMTUD is the need for positive feedback.

Positive feedback to the original source based on actual data
packets, yes; not to the tunnel ingress.

> received. The sample size depends on the flow rate.

The requirement is to discover the lowest common denominator
MTU among all paths that contribute to the multipath. Sampling
doesn't do that, because it only tells about the packets that got
through and doesn't tell about the packets that *didn't* get through.
So, a large packet that successfully traverses one path in the multipath
does not reveal the instances of smaller packets that cannot traverse
a different path in the multipath.

> sets vary by a property other than packet length.

If you mean to say that the ingress needs to wait for an ACK and then
consider the probe lost if the ACK doesn't come back in X seconds, then
the ingress design is already way too complex in that it needs to maintain
state, timers, etc.

> past packets as implied success of future packets.

But, when it is the original source doing PLMTUD the probes and
data packets all look exactly the same to the ECMP/LAG equipment
so they necessarily travel over the same path.

> of past packets predicts that future packets will also succeed.

With PMTUD, *all* packets are probes. Negative feedback therefore
is useful when it becomes available, but when PTBs are lost the source
will still be doing PLMTUD and will recover.

> *know* otherwise (i.e., you've deployed the network).

Have the tunnel ingress use PMTUD for packets larger than 1500, and
have the original source use PLMTUD for packets larger than 1500 is
the way to get to larger MTUs.

> generates the IP source and destination based on the ingress and egress.

Sorry; dyslexia - meant to say RFC2983.

> per-flow basis (via sampling).

Again, sampling only tells about the packets that did get through; not
the packets that didn't get through. And there is no way of knowing
when all paths of the multipath have been sampled.

> You only have to (statistically) track the packets you're already sending.

Tracking packets that have already been sent is too much state
for the ingress, and doesn't scale if there are many egress'.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Wed, 19 Aug 2015 17:15:15 -0700

On 8/19/2015 4:27 PM, Templin, Fred L wrote:
...
> packets larger than 1500.

Why 1500? Packets in the range 1500-ENCAPS+1..1500 might exceed the
egress reassembly capability and could be black-holed even if fragmented.

However, regardless, you cannot *require* sources that use tunnels to
use PLMTUD. Either a tunnel supports IPv6 or it does not; there is no
way to specify a tunnel to support "only flows that use PLMTUD", esp.
because the tunnel cannot enforce that requirement.

> packets, yes; not to the tunnel ingress.

PLMTUD sends positive feedback to the *packetization layer* (by
definition). The IP fragmentation of the ingress encapsulator is a
packetization layer, which is why it is an obvious use of PLMTUD.

Sure, it's great of the source runs PLMTUD but that won't ever "tune" to
the packetization of the ingress fragmentation.

> MTU among all paths that contribute to the multipath. 

...for that flow.

> Sampling

Not all packets are probes - the probes are a "sample" of the flow.

Probes never tell you about what happens to non-probes. That's now
PLMTUD works.

> a different path in the multipath.

That's true for all uses of PLMTUD. Probes don't tell you about non-probes.

> the ingress design is already way too complex in that it needs to maintain

That's already a part of many tunnel mechanisms to detect when the
tunnel goes down.

> so they necessarily travel over the same path.

An ingress can use the existing traffic as "probes"; a "probe" is a
packet that isn't fragmented.

> will still be doing PLMTUD and will recover.

We're going around in circles on this point - PMTUD works when it works
and cannot be used when it fails and the sources don't use PLMTUD.

It's not realistic to have this document declare that tunnels work only
for sources that support PLMTUD.

So we have to have another approach.

> the way to get to larger MTUs.

That might work but there's no way to enforce that.

> Sorry; dyslexia - meant to say RFC2983.

Then same solution as for RFC6438.

> when all paths of the multipath have been sampled.

That's already true for all PLMTUD uses, even from the source to the
destination without any tunnels.

> for the ingress, and doesn't scale if there are many egress'.

The alternative is to have a requirement that isn't enforceable and
isn't deployable.

Tunnels are work. If they create work for the rest of the Internet, then
they're not tenable as part of the Internet.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Aug 2015 15:55:55 +0000

Hi Joe,

> Sent: Wednesday, August 19, 2015 5:15 PM

I thought we already got past this. All of the IPv6-in-IPv4 tunneling RFCs
are proof that it is possible to publish standards that exceed the IP
minimum EMTU_R. What we are asking for essentially amounts to the
same thing, i.e., tunnels that cannot directly use standard PMTUD
need to be able to reassemble (1500+ENCAPS) at a minimum.

> because the tunnel cannot enforce that requirement.

Remember, the tunnel does the work necessary to take care of
all packets up to 1500 bytes, and expects the source to do some
RFC4821 work if it wants to send larger packets.

> packetization layer, which is why it is an obvious use of PLMTUD.

I used to think that - and my drafts even said that - up until I found
out about the multipath problem. If there is no way of knowing
whether the probe packets will take the same path as the data
packets, PLMTUD will not get the job done.

> the packetization of the ingress fragmentation.

No tuning, and the arrangement is very simple. The source can expect
that packets up to 1500 get through or a suitable PTB message returned.
For packets larger than 1500, it should really use PLMTUD.

> > MTU among all paths that contribute to the multipath.

So, you want to have the ingress determine a per-flow MTU? That is
a much more ambitious goal than determining a lowest common
denominator MTU for all flows. Remember also that the ingress
can aggregate many flows from many sources, and has no way
of telling what paths the packets will take when they enter the
tunnel. 

> packet that isn't fragmented.

Putting a probe bit on a data packet is certainly an alternative AERO and
SEAL have suggested in the past. The trouble is that the ingress has no
way of knowing which paths will be traversed once the packet enters
the tunnel.

One alternative I thought of a long time back was to try to rig the tunnel
to provide limited multipath. By limited, I mean aggregate all of the flows
sent by original sources into just a few flows between the ingress and
egress through some sort of hashing. So, for example, allow only four
flow label values in the encapsulation header and all source flows are
hashed into one of the four encapsulation flow labels. That way, the
egress would only need to test four paths - not thousands or more.

That would be fine, but if the ECMP equipment inside the tunnel did
DPI then no amount of rigging at the ingress will prevent data packets
from following a different path than the probe packets.

> and cannot be used when it fails and the sources don't use PLMTUD.

That is why the tunnel takes care of all packets up to 1500 bytes and
lets the original source take care of larger packets. Original sources
just aren't sending jumbos (anything above 1500) to Internet
destinations today. Asking them to do RFC4821 if they ever decide
to start sending jumbos doesn't seem like an unreasonable ask.         

> for sources that support PLMTUD.

This lets tunnels work for all source that send packets no larger than
1500. For larger packets, the source has to have a stake in the game
to make sure that its large packets get through.

> That might work but there's no way to enforce that.

If the original source sends packets larger than 1500 but without using
RFC4821, then there is a reasonable likelihood that the ingress will
still return a suitable PTB message if the packet is lost the same as
for any link. So, the tunnel still honors standard PMTUD.

> destination without any tunnels.

No, with PLMTUD the data packets and probe packets look the same all
the way to the core. So, ECMP/Lag would always place both the data
packets and probe packets on the same path.

> they're not tenable as part of the Internet.

Tunnels should get packets up to 1500 through even if they need to do
a little work. Hosts that send packets larger than 1500 need to do work
whether or not there is a tunnel in the path.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Aug 2015 09:17:21 -0700

On 8/20/2015 8:55 AM, Templin, Fred L wrote:
...
> minimum EMTU_R.

Those are standards-track; this is not. As informational or even BCP, we
can point out where existing standards are inconsistent with each other
but we can't establish new requirements.

> RFC4821 work if it wants to send larger packets.

First, the tunnel would do the work for TTPs up to 1500 bytes only if we
could assume that the egress can reassemble 1500+ENCAPS. That's why I
used 1500-ENCAPS as the limit of what we know the tunnel can transit.

Forwarding larger packets when we don't know they can get there
encourages black holes for those not using PLMTUD. The alternative is to
block those packets unless the ingress knows they can get to the egress.

You're trading efficiency for capability, and I am not. Until sources
are required to use PLMTUD for larger packets, it seems irresponsible
for a link to accept packets unless it knows they can transit the link.

> packets, PLMTUD will not get the job done.

Agreed. That's why the obvious solution is a combination of the two:

	- if the ingress-egress implements PLMTUD between the two,
	or some other feedback mechanism, then larger TTPs can
	be allowed to transit the tunnel as a probe

	- if there is no ingress-egress PLMTUD or other direct
	coordination, then larger TTPs MUST be rejected with
	ICMP PTB messages

> For packets larger than 1500, it should really use PLMTUD.

We cannot control the source. We can indicate requirements for the ingress.

...
> So, you want to have the ingress determine a per-flow MTU? 

It would have to...

> That is
> denominator MTU for all flows. 

...to determine the lowest common denominator.

> Remember also that the ingress

The ingress has to track what happens to different packets - and by
doing so it would learn about different paths taken.

...
> way of knowing which paths will be traversed once the packet enters

That's why the probe bit has to be set on each flow.

> egress through some sort of hashing.

That's another option that would help limit the number of flows that
need to be sampled. It goes outside existing "copy the flow ID"
requirements, but would be a good way to reduce ingress-egress PLMTUD
overhead.

...
> from following a different path than the probe packets.

Put another way, it would never prevent the need to probe different TTP
flows, yes.

...
> to start sending jumbos doesn't seem like an unreasonable ask.         

Jumbograms have a definition; let's please not reuse that term for
packets over 1500 bytes.

Strictly, hosts that send packets larger than 1280 already need to
expect to manage the path MTU themselves.

The intarea-tunnels rules already work up to 1500-ENCAPS. There's no way
to make them work for packets in the range 1500-ENCAPS+1...1500 without
requiring the egress to reassemble fragments that are larger than the
current minimums AND have the ingress be able to trust that this is true.

> for any link.

We can declare that the ingress send the PTB when the TTP>1500 packet
arrives. But we cannot expect that the ingress-egress path will return
an ICMP error - that's why PLTMUD exists.

> So, the tunnel still honors standard PMTUD.

The tunnel is over the Internet.

The source-dest traffic is over the Internet.

If the source-dest traffic needs to use PLMTUD, then so do the
ingress-egress.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Aug 2015 17:25:59 +0000

Hi Joe,

> Sent: Thursday, August 20, 2015 9:17 AM

No, it is not a standards-track but it gives guidance to designers of
standards-track documents. If it can't at least do that, what good is it?

> used 1500-ENCAPS as the limit of what we know the tunnel can transit.

Again, 1500 is only for IPv6. For IPv4, the only guarantee is 576. And,
many tunnel docs have ignored the specs and declared a minimum
EMTU_R of their own choosing.

> block those packets unless the ingress knows they can get to the egress.

That part disturbs me. Setting an initial small MTU until it is proven that
a larger MTU can work means that hosts will initially get a too-conservative
MTU estimate and may or may not try again later to discover that larger
packets are possible. Setting an indefinite MTU (ala RFC2473) means that
sources can begin using large packets immediately.

> for a link to accept packets unless it knows they can transit the link.

It is a link that traverses a path. There are path characteristics within
that path. One of the characteristics is the largest packet size that
can traverse the path without loss due to a size restriction.

When a large packet enters the tunnel, it may be lost with a PTB
message returned. The ingress can then reflect that size restriction
back to the original source the same as for any IP path. The tunnel
therefore does not make standard PMTUD any less brittle than it
already is; the tunnel is simply a link configured over a sub-path
of the end-to-end path.

> 	be allowed to transit the tunnel as a probe

There is still no way of knowing whether all paths in a multi-path
have been tested.

> 	coordination, then larger TTPs MUST be rejected with

Not true; if there is no ingress-egress PLMTUD the ingress can
still admit packets larger than 1500 and return ICMP PTB messages
if it discovers that the size is too large. The ingress is no less likely
to receive ICMP PTB messages than the original source is. So,
standard PMTUD is no less robust with a tunnel that does this
in the path than without the tunnel in the path.

Again, the size threshold for this is 1500 bytes; the tunnel takes care
of all packets that are no larger than 1500 and ensures that they are
delivered to the egress even if a little bit of fragmentation is needed. 

> We cannot control the source. We can indicate requirements for the ingress.

The source should always use PLMTUD for packets larger than
1500 bytes. This is true in all cases but especially true when there
is a tunnel in the path. So, this document seems like a good place
to update RFC1122 and RFC6434.

> ...to determine the lowest common denominator.

There is no way of knowing whether you have tested all paths
in a multipath.

> doing so it would learn about different paths taken.

There is no way of knowing whether you have tested all paths
in a multipath.

> That's why the probe bit has to be set on each flow.

How does the ingress know what constitutes a flow? And,
if the ingress' idea of what constitutes a flow does not
match the ECMP/LAG equipment's idea of what constitutes
a flow then there is opportunity for probe packets and data
packets to travel different paths.

> requirements, but would be a good way to reduce ingress-egress PLMTUD

True if it worked. But, it doesn't work.

> Jumbograms have a definition; let's please not reuse that term for

There are already many definitions of jumbo "IPv6 jumbo", "Gig-E
jumbo", "baby jumbo", etc. But, I don't mind not using the term here.

> expect to manage the path MTU themselves.

But, there are just too many nodes out there that expect 1500s
to get through or a suitable PTB message returned, and no
recommendation we make here will cause them to change.

> current minimums AND have the ingress be able to trust that this is true.

The recommendation is that tunnels SHOULD reassemble at least
1500+ENCAPS, so they SHOULD reassemble at least 2KB for safe
measure. Tunnels that can't do this are not broken; it's just that
they won't be able to take advantage of larger sizes when they
are available and they might exacerbate ICMP black holing
because of configuring a degenerate MTU.

> If the source-dest traffic needs to use PLMTUD, then so do the

Disagree. If the tunnel can do at least 1500 using a small amount of
fragmentation if necessary, then it doesn't need to use PLMTUD.

That said, if the tunnel had some way of probing for (1500+ENCAPS)
then it might be able to discontinue fragmentation and begin sending
whole packets. But, such probing would be subject to the same sort
of multipath issues we have been discussing in these past several
messages.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Aug 2015 10:32:29 -0700

> There is no way of knowing whether you have tested all paths

Let's please cut to the chase on this point.

If you test every flow ID that you see, then you will be testing as much
as the source application PLMTUD tests.

If there is additional multipath within a flow, then nothing will test
all possible paths, yes. Nothing includes PLMTUD, PTMUD, and the network
manager who thinks they know what's deployed.

So, ultimately, if the egress sees a flow and tells the ingress, you've
tested exactly what you need to over the tunnel - exactly what PLMTUD is
designed to - exactly what the source E2E PLMTUD can ever test.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Aug 2015 17:59:39 +0000

Hi Joe,

> Sent: Thursday, August 20, 2015 10:32 AM

How can you test every flow that you see? The ingress may aggregate
the flows of potentially millions of original sources.  And, it is not just
about IPv6 flow label. It is also about source and destination address,
Traffic Class, IP protocol number, possibly port numbers, possibly also
application data content, etc. Then, what ECMP/LAG equipment may
see as a flow may be different than what the ingress sees as a flow. 

> all possible paths, yes. Nothing includes PLMTUD,

PLMTUD, when used from the source, uses data packets that
are part of the flow as probes. It is the best that can be done,
and the original source is the only actor in a position to do that.

> PTMUD, and the network

PMTUD is an exception because it tests *every* packet, and not just
an occasional probe. That PTB messages can be lost sometimes is
unfortunate, but the host using PLMTUD for packets larger than
1500 takes care of that case.

> designed to - exactly what the source E2E PLMTUD can ever test.

The ingress/egress would need to test all flows, and not just one
or two flows. And, there is no way for the ingress to know whether
its notion of a flow matches with the network's notion of a flow. 

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 20 Aug 2015 11:39:25 -0700

I'm answering in parts with specific subject lines to make these threads
easier to track:

Regarding setting the ingress MTU to 1500-ENCAPS:

> sources can begin using large packets immediately.

The tunnel always has an MTU - the size it *knows* can transit. Just
like any other L2 interface.

PLMTUD and PMTUD can ignore known path or interface MTUs where desired,
but if they are directly connected, should never send a packet larger
than the
next interface can accept. That's exactly how routers know when to send
an ICMP PTB back.
Treating the tunnel as having an indefinite MTU and hoping that PMTUD
works - or that the source uses PLMTUD - is a recipe for black holes.

An ingress-egress pair can determine a larger MTU and reconfigure the
"ingress as interface" MTU accordingly. That setting will work for
subsequent tests, whether for current connections or new connections.

This how the rest of the Internet works. I see no reason to treat
tunnels differently.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 20 Aug 2015 19:01:00 +0000

Hi Joe,

> Sent: Thursday, August 20, 2015 11:39 AM
> Subject: starting with a conservative MTU
> 

It is no worse than PMTUD already is in the Internet at large today.
It does not introduce any new black holes than what were there
already. It says that sources that send packets larger than 1500
w/o explicitly checking to make sure they get through are asking
for trouble whether or not there is a tunnel.

> This how the rest of the Internet works. I see no reason to treat

A variable MTU that starts small and grows larger? That is not how
links work. Then, we get back to the subject of PLMTUD probing in
the presence of multipath and the whole thing becomes untenable.

Thanks - Fred