Subject: [v6ops] Proxy function for PTB messages on the tunnel end

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 22 Mar 2021 11:28:07 +0000

Hi Joseph,
I probably need to tell why I like the initial RFC 2473: it requests PTB “proxy” functionality from tunnel ends. (well, it is not called “proxy” inside RFC 2473 – it just discusses how to re-create PTB from PTB on the tunnel end). This initial architecture decision of IPv6 (to inform the real traffic source) is the basement for PMTU to work. It is better to fix it, not to invent some other patches for MTU discovery.
Longer explanation:
If PTB message would be created on the tunnel path – it would easily inform “tunnel end” of Oversized packet – tunnel MTU could be decreased.
It is already a solution because the next Oversized packet from the source would get PTB response from the tunnel end itself – the source would get PTB after the second oversized packet.
But it was better to inform the real traffic source after 1st Oversized packet. Hence, PTB proxy on the tunnel end is better.

I am not ready to discuss all corrections of draft-ietf-intarea-tunnels section 5.2 to RFC 2473 – they do not have relationships to PMTUD.
Except one that would lead to the massive fragmentation that I would discuss in the next email.
Eduard
From: Joseph Touch [mailto:[email-masked]]
Sent: Sunday, March 21, 2021 7:23 PM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>
Subject: Re: [v6ops] draft-vasilenko-v6ops-ipv6-oversized-analysis-00

Hi, all,

Spoiler alert if you don’t want to read the whole post:
            - draft-vasilenko makes erroneous claims as to the content in draft-tunnels
            - draft-tunnels and draft-vasilenko are consistent (once the latter is corrected) in their mutual conclusions
                        - draft-tunnels on the need for fragmentation over finite MTU paths
                        - draft-vasilenko in encouraging increases in those finite MTUs

Joe

---

First, draft-ietf-intarea-tunnels is discussed on the int-area list; after review of the information below, if you still believe there are issues to be addressed in that doc, you should post them there.

The technical errors in RFC2473 have been indicated in that document since draft-ietf-intarea-tunnels-01, posted in July 2015. They remain accurate, IMO.

Note that I ceased performing in-place updates of that document because of *lack of active discussion* and because in-place updates are a waste of my time.

I am glad to see someone in IPv6 interested now, and would be glad to update my draft as needed.

FWIW, having read your doc, here are its errors in misstating the content of my draft:

- your doc mistakenly assumes that mine requires IPv6 hosts to send 1500B packets if they can, even if tunnels are on the path
            as with any IPv6 path, the source should send fragments no larger than the entire path can transit, whose reassembled size is no larger than the receiver can reassemble
            those original fragments are what enter the on-path tunnels, so they should be no larger than the tunnel egress can fragment
            and those original fragments would be encapsulated and then source fragmented by the tunnel according to the same (recursive) policy

- nothing in draft-tunnels assumes ICMP PTB cannot adjust these sizes or that the tunnel cannot use PLPMTUD
            see sec 4.3.1 of v10

- draft-tunnels does not “introduce” a new variable called tunnel MTU; I introduced the terminology, but the concept is as old as tunnels
            I coined that term to refer to the MTU across the tunnel with reassembly at egress (which already exists), as different from the MTU between ingress and egress (which I call tunnel MAP)
            sec 4.2.3 of v10 doesn’t claim this value cannot be set; in explains that PMTUD has no role in discovering its value:

          Note, however, that PMTUD never discovers

          EMTU_R that is larger than the required minimum; that information is

          available to some upper layer protocols, such as TCP [RFC1122<https://tools.ietf.org/html/rfc1122>], but

          cannot be determined at the IP layer.
            I never said it cannot be discovered
                        it should be (e.g., by a tunnel configuration protocol)
                        note that there are no current protocols that do this, even without tunnels (i.e., discover larger EMTU_R)
                        I can add that point as clarification

- draft-tunnels does not increase IPv6 fragmentation
            please indicate why you believe it would (notably here "a considerable increase in fragmentation is proposed for the reasons of academic purity”)

- draft-tunnels does not claim fragmentation is the only solution to oversize packets
            it addresses how and where to handle tunnels in the presence of packet limits, of which path MTU is only one

- ICMP PTB is not a solution out to the origin source
            that would potentially drop the IPv6 path MTU below 1280, given enough tunnel overhead (or layers thereof), a violation of IPv6
            so yes, in that case, the ONLY solution that preserves IPv6 in the presence of tunnels with that much overhead would be ingress source fragmentation

- sec 3.3 of my doc DOES allow ICMPs to be relayed back to the source
            it merely states that they should be generated when a packet too large to ingress arrives,
            NOT when an internal tunnel ICMP is received by the ingress

            the point is that the origin source sees the ingress as a router on the path,
            so it should get ICMPs from that router only when packets arrive at that router, not when its tunnel fails downstream

            this makes ICMP relay *easier* and more reliable to implement; the ingress gets tunnel ICMPs to learn the tunnel’s effective link MTU,
            then uses that link MTU to send ICMPs back

            yes, this is to allow the tunnel to act as the link *that it is*, but it does not prohibit ICMP info from flowing back to the source

And finally:

- nobody is claiming we shouldn’t increase link MTU
            draft-tunnels would still be relevant, no matter how large the MTU is, for the reasons I state in that doc

One other observation:

- your statistics for fragment drops apply only when the fragment is visible to the IP layer
            there are intermediate layers that hide fragmentation for exactly this reason, e.g., UDP tunnels, GRE, etc.

---

On Mar 21, 2021, at 1:59 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Dear Experts,
I have seen many recent activities in IETF related to MTU problems. Well, maybe not so active as some others, but active anyway. Many other active drafts are evaluated in this draft.
I had an idea what is the right way to solve problems in this area, but after the research, it has been found that foundations were discussed in RFC 2473 (Dec 1998). Just people have forgotten about it.
We have discussed it with co-authors and we have decided that it make sense to publish the research because it looks at the problem in a systematic approach.

The one thing that is alarming in this research: draft-ietf-intarea-tunnels is pushing for much more fragmentation for pure Academic reasons. This draft is already referenced by many other documents.
I believe that not many people have spent enough time to understand it's complexity to reveal the truth: the majority of the IPv6 traffic would be fragmented if it would follow draft-ietf-intarea-tunnels.

Thanks to everybody who would spend enough time to produce comments.
Eduard
-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Friday, March 19, 2021 11:07 PM
To: Dmitriy Khaustov <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt

A new version of I-D, draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt
has been successfully submitted by Eduard Vasilenko and posted to the IETF repository.

Name:             draft-vasilenko-v6ops-ipv6-oversized-analysis
Revision:         00
Title:               IPv6 Oversized Packets Analysis
Document date:          2021-03-19
Group:             Individual Submission
Pages:              19
URL:            https://www.ietf.org/archive/id/draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt
Status:         https://datatracker.ietf.org/doc/draft-vasilenko-v6ops-ipv6-oversized-analysis/
Htmlized:       https://datatracker.ietf.org/doc/html/draft-vasilenko-v6ops-ipv6-oversized-analysis
Htmlized:       https://tools.ietf.org/html/draft-vasilenko-v6ops-ipv6-oversized-analysis-00

Abstract:
  The IETF has many new initiatives relying on IPv6 Enhanced Headers
  added in transit: SRv6, SFC, BIERv6, iOAM. Additionally, some recent
  developments are overlays (SRv6, VxLAN) over IPv6. It could create
  oversized packets that need to be dealt with. This document analyzes
  available standards for the resolution of oversized packet drops.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

_______________________________________________
v6ops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/v6ops

---

From: Joseph Touch <[email-masked]>
Date: Mon, 22 Mar 2021 08:28:44 -0700

Hi, Eduard,

> On Mar 22, 2021, at 4:28 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

You should review the discussions on why PMTUD does not work in the Internet. That’s why we have PLPMTUD.

So I’m not clear that fixing PMTUD is worth any effort at all. 

However...

> If PTB message would be created on the tunnel path – it would easily inform “tunnel end” of Oversized packet – tunnel MTU could be decreased.

Path MTU ICMP errors indicate when a packet CANNOT TRAVERSE A LINK. 

As a *link*, a tunnel’s MTU is its EMTU_R. It cannot be its EMTU_S or path MTU. If it were, then there could never be IPv6-in-IPv6 because no IPv6-in-IPv6 tunnel can relay internal segments of 1280B without ingress source fragmentation. 

Is that what you want? 

> From: Joseph Touch [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>; int-area <[email-masked] <mailto:[email-masked]>>
>           available to some upper layer protocols, such as TCP [RFC1122 <https://tools.ietf.org/html/rfc1122>], but
> On Mar 21, 2021, at 1:59 AM, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Dear Experts,
> From: [email-masked] <mailto:[email-masked]> [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Dmitriy Khaustov <[email-masked] <mailto:[email-masked]>>; Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>; Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>; Xipengxiao <[email-masked] <mailto:[email-masked]>>; Xipengxiao <[email-masked] <mailto:[email-masked]>>
> URL:            https://www.ietf.org/archive/id/draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt <https://www.ietf.org/archive/id/draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt>
> Status:         https://datatracker.ietf.org/doc/draft-vasilenko-v6ops-ipv6-oversized-analysis/ <https://datatracker.ietf.org/doc/draft-vasilenko-v6ops-ipv6-oversized-analysis/>
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-vasilenko-v6ops-ipv6-oversized-analysis <https://datatracker.ietf.org/doc/html/draft-vasilenko-v6ops-ipv6-oversized-analysis>
> Htmlized:       https://tools.ietf.org/html/draft-vasilenko-v6ops-ipv6-oversized-analysis-00 <https://tools.ietf.org/html/draft-vasilenko-v6ops-ipv6-oversized-analysis-00>
> 
> 
> Abstract:
> Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org <http://tools.ietf.org/>.
> 
> The IETF Secretariat
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/listinfo/v6ops>

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 22 Mar 2021 16:54:41 +0000

Hi Joseph,

I believe that draft-ietf-intarea-tunnels is Academic purification for the reason that I do not understand.
Many new names disturb a lot – I have spent more time for draft-ietf-intarea-tunnels than for any other to understand. This draft looks for me as over-engineering.
Moreover, in my draft, I believe that the paragraph about draft-ietf-intarea-tunnels is very difficult to understand. But I have not found how to make it simple because I need to somehow reflect the tremendous complexity of draft-ietf-intarea-tunnels. May be would find later how to explain it shorter – I am not happy about this paragraph.

You are not just explaining something to somebody. You are trying to change. It should have the reason - some motivation.
What benefit anybody would have if every virtual link would have 2 MTUs? Especially if the biggest one is not managed automatically.

Eduard
From: Joseph Touch [mailto:[email-masked]]
Sent: Monday, March 22, 2021 6:29 PM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>
Subject: Re: Proxy function for PTB messages on the tunnel end

Hi, Eduard,

On Mar 22, 2021, at 4:28 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

However...

Longer explanation:

Is that what you want?

I am not ready to discuss all corrections of draft-ietf-intarea-tunnels section 5.2 to RFC 2473 – they do not have relationships to PMTUD.
Except one that would lead to the massive fragmentation that I would discuss in the next email.
Eduard
From: Joseph Touch [mailto:[email-masked]]
Sent: Sunday, March 21, 2021 7:23 PM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; int-area <[email-masked]<mailto:[email-masked]>>
Subject: Re: [v6ops] draft-vasilenko-v6ops-ipv6-oversized-analysis-00

Hi, all,

Joe

---

And finally:

One other observation:

---

Dear Experts,

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Friday, March 19, 2021 11:07 PM
To: Dmitriy Khaustov <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt

Abstract:

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org/>.

The IETF Secretariat

---

From: Joseph Touch <[email-masked]>
Date: Mon, 22 Mar 2021 10:33:32 -0700

Eduard,

> On Mar 22, 2021, at 9:54 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,
>  

It is an attempt to try to explain a complex topic that many have confused using different meanings for the same term. It evolved over nearly a decade to the terms inside and was discussed *extensively* in int-area.

> Many new names disturb a lot – I have spent more time for draft-ietf-intarea-tunnels than for any other to understand. This draft looks for me as over-engineering.

I encourage you to examine the extensive discussion in int-area in the mail archives.

> What benefit anybody would have if every virtual link would have 2 MTUs? Especially if the biggest one is not managed automatically.

I didn’t create that situation - it already exists.

The issue is that ICMP tells you when a path MTU changes, but there is no ICMP that tells you about EMTU_R. Nor is there an ICMP that says “hey, you CAN send 1500 bytes but if you send smaller it’d be better”. ICMP PTB (in IPv6, and the corresponding one for IPv4) tell you when a packet CANNOT cross a path.

A tunnel that must source-fragment to support required IPv6 MTUs (e.g.,IPv6-in-IPv6 over a 1280B path) CAN send packets up to 1500B across a tunnel. It is an error for that tunnel to relay a PTB message that says “cannot support 1280B” just because it has to be fragmented.

You are confusing PTB with “packet bigger than I’d like, but will still get there”. 

We have no ICMP message for that. And creating new ICMP messages is a waste of time, given how they’re already filtered extensively.

So what is your solution? I agree that making MTUs bigger would help, but they never get around this problem.

Joe

> On Mar 22, 2021, at 4:28 AM, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Joseph,

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 22 Mar 2021 19:32:55 +0000

Hi Joseph,
I believe that the second MTU parameter for the tunnel interface has been created exclusively in the draft-ietf-intarea-tunnels. It did not exist before.
Router has to deal with EMTU_R only in the control plane (where LINUX is opening TCP connections).
I did not search specifically, but I’ve read before tunneling specifications – the data plane operates only with one tunneling MTU parameter. It was enough for the last 30 years.
Please, show me any tunneling specification (GRE? VxLAN? L2TPv3? whatever) where discussion exists about 2 MTUs.
Please, show me the URL to the documentation of any vendor who has 2 MTU parameters for the tunnel interface. It is very interesting how they explain the need for the second one.

I could imagine for the second MTU only as of the buffer size in the particular vendor implementation (probably with fragmentation case).
Are you aware of any situation when the vendor did not manage their buffers properly for us to intrude into the situation?
It is probably not the reason anyway if others did manage it properly. They should keep buffer much bigger anyway, because reordering and packets jitter may request a lot of memory.

What to do if one could not push the traffic source to decrease MTU because it is already 1280 (minimum)?
RFC 2473 said in 1998: it is the only valid reason to fragment. Many other RFCs do not care about this corner case at all (still not a lot of complains because 220B is available for all additional headers).
I believe it is better to return to RFC 2473 for the case when the packet is already 1280 and could not be smaller.

Eduard
Sent: Monday, March 22, 2021 8:34 PM

Eduard,

On Mar 22, 2021, at 9:54 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Joe

Eduard

Hi, Eduard,

Hi Joseph,

However...

Longer explanation:

Is that what you want?

Hi, all,

Joe

---

And finally:

One other observation:

---

Dear Experts,

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Friday, March 19, 2021 11:07 PM
To: Dmitriy Khaustov <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt

Abstract:

The IETF Secretariat

---

From: Joseph Touch <[email-masked]>
Date: Mon, 22 Mar 2021 12:48:30 -0700

> On Mar 22, 2021, at 12:32 PM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

Tunnels have always had both the MTU that isn’t defragmented between in ingress and egress and an EMTU_R at the egress.

In some protocols (e.g., that have no ingress source fragmentation), the two are the same. But they’ve always been there.

> Router has to deal with EMTU_R only in the control plane (where LINUX is opening TCP connections).

Hosts deal with EMTU_R; routers do not (unless acting as a host, i.e., as an IP source/sink).

> I did not search specifically, but I’ve read before tunneling specifications – the data plane operates only with one tunneling MTU parameter. It was enough for the last 30 years.

IP tunes only the path MTU; it relies on the transport protocols to tune EMTU_R (and not all transports do).

That doesn’t mean EMTU_R didn’t exist or isn’t relevant. It’s just *ignored*.

> Please, show me any tunneling specification (GRE? VxLAN? L2TPv3? whatever) where discussion exists about 2 MTUs.

Most don’t - again, that’s an error, not a feature. 

Most tunneling systems incorrectly confuse the path MTU and EMTU_R of the tunnel as either equal (which they sometimes are) or ignore the latter.

> Please, show me the URL to the documentation of any vendor who has 2 MTU parameters for the tunnel interface. It is very interesting how they explain the need for the second one.

The point of draft-tunnels is to point out this flaw. Vendors get this wrong, which is WHY TUNNELS BREAK.

> I could imagine for the second MTU only as of the buffer size in the particular vendor implementation (probably with fragmentation case).

For IPv6 tunnels, it is *defined in the IP spec* as of RFC2460 as 1500B, as distinct from the path of 1280B

> Are you aware of any situation when the vendor did not manage their buffers properly for us to intrude into the situation?

The vendors incorrectly relay PTBs and things break; that’s the situation we all live with today. That’s why you’re trying to make MTUs bigger and I’m trying to get tunnels implemented correctly.

> It is probably not the reason anyway if others did manage it properly. They should keep buffer much bigger anyway, because reordering and packets jitter may request a lot of memory.

Having a bigger buffer will do NO good unless the source knows about it *and uses it*.

> RFC 2473 said in 1998: it is the only valid reason to fragment. Many other RFCs do not care about this corner case at all (still not a lot of complains because 220B is available for all additional…

It’s not a corner case; it’s *exactly* why most implementations don’t use 1500B packets either; they drop down to 1400 or so.

> I believe it is better to return to RFC 2473 for the case when the packet is already 1280 and could not be smaller.

RFC2473 would fail to support IPv6 over IPv6 if it relays PTBs from inside the tunnel, as it is required to do.

Joe

> On Mar 22, 2021, at 9:54 AM, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Joseph,
>

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 22 Mar 2021 20:33:11 +0000

Hi Joseph,
You insist that the second MTU restriction exists for all data plane implementations, just it is not visible and not discussed: neither in any RFC nor in any vendor documentation.
But then should be the same effect: fragmentation between MTUs.
I am sure that it is not the case.
PTB would decrease the only MTU that tunnel has,
The next oversized packet would get PTB message in the direction of the host.

The buffer is not relevant for the discussion because data plane firmware just does not have a second MTU. This restriction just does not exist in the real life on the planet Earth.
Eduard
From: Joseph Touch [mailto:[email-masked]]
Sent: Monday, March 22, 2021 10:49 PM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>
Subject: Re: Proxy function for PTB messages on the tunnel end

On Mar 22, 2021, at 12:32 PM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Joe

Eduard

Eduard,

Hi Joseph,

Joe

Eduard

Hi, Eduard,

Hi Joseph,

However...

Longer explanation:

Is that what you want?

Hi, all,

Joe

---

And finally:

One other observation:

---

Dear Experts,

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Friday, March 19, 2021 11:07 PM
To: Dmitriy Khaustov <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>; Xipengxiao <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-vasilenko-v6ops-ipv6-oversized-analysis-00.txt

Abstract:

The IETF Secretariat

---

From: Joseph Touch <[email-masked]>
Date: Mon, 22 Mar 2021 14:03:42 -0700

> On Mar 22, 2021, at 1:33 PM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

No; it is DEFINED in RFC1122 and noted in both RFC791 and RFC2460 (and 8200). It’s exactly the difference between what can be sent on a path and what is reassembled at an endpoint. Ingress are sources and egresses are sinks; host requirements apply to BOTH.

> But then should be the same effect: fragmentation between MTUs.

They do not.

> The buffer is not relevant for the discussion because data plane firmware just does not have a second MTU. This restriction just does not exist in the real life on the planet Earth.

PTB inside a tunnel needs to update the ingress to change fragment size, because it says “the tunnel path can’t handle the packet size being relayed”, but the path CAN handle the packet if the ingress adjusts. That’s why it is INCORRECT to relay that info to the origin source - PTB inside the tunnel does not mean the tunnel has failed, but rather that it needs to be adjusted.

When the egress cannot reassemble a packet because it is too large, it should send some sort of error back - such as EMTU_R exceeded, but *there is no such ICMP error*. However, IF THERE WERE such a message, then THAT message inside the tunnel should generate a PTB to the origin host. That’s because if you exceed EMTU_R of the tunnel egress, then the source has to send smaller packets - there’s no other way.

In a nutshell, THAT IS WHY THEY ARE DIFFERENT:
	- PTB inside the tunnel means “I’d PREFER a different size, but can still live with this size”
		it cannot be relayed to the origin because there is no meaning that indicates PREFERENCE - only strict capability
	- EMTU_R exceeded inside the tunnel means “I cannot accommodate this size at all”
		which WOULD need to be relayed to the origin source as PTB

The ultimate problem here is that:
	- there aren’t the right ICMP messages to do this correctly
	- there’s no point in creating new ICMP messages to fix the issue, given ICMPs are generally blocked

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 23 Mar 2021 13:21:06 +1300

On 23-Mar-21 08:32, Vasilenko Eduard wrote:
...
> What to do if one could not push the traffic source to decrease MTU because it is already 1280 (minimum)?

Nothing. Any tunnel MUST do whatever it has to do to carry a 1280 byte packet. That is by definition always the tunnel's problem and never the host's problem.

   Brian

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 23 Mar 2021 09:18:15 +0000

Hi Joseph,
I still do not understand why you believe that any tunnel has a second MTU restriction now.
The rest of your logic is based on this assumption that I do understand how you come to it. But the rest does not make sense to discuss if the initial assumption is not right.

1.       It could not be the case for tunneling that does not support fragmentation (L2TPv3, MPLS, VxLAN). Fragmentation buffer is just not needed in principle. Second restriction is not needed in principle.

2.       Reassembly buffer is needed for tunneling that does support fragmentation (NVO3, IPv6 GRE). But under no circumstances it should have any relationship to EMTU_R developed for TCP stack on the host. Vendors have chosen this fragmentation buffer always much bigger than maximum MTU for not to have another reason for fragmentation.
That’s it – no second MTU restriction to pay attention to. It is the reason why this is not documented in any vendor’s documentation: no problem – no need to document.
But you have assumed that vendors did always have fragmentation buffer with limited size related to EMTU_R. It is just not the case. Data Plane has no TCP stack to inherit TCP restrictions.

Again: the current standards and implementations have only 1 MTU restriction per tunnel. It does permit to inform the host about oversized packets (by PTB).
draft-ietf-intarea-tunnels proposes new solutions with 2 MTU restrictions per tunnel (small/restricted reassembly buffer size and MTU on the tunnel path) that pushes for fragmentation between these restrictions.
Ed/
From: Joseph Touch [mailto:[email-masked]]
Sent: Tuesday, March 23, 2021 12:04 AM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>
Subject: Re: Proxy function for PTB messages on the tunnel end

On Mar 22, 2021, at 1:33 PM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

They do not.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Tue, 23 Mar 2021 07:59:07 -0700

> On Mar 23, 2021, at 2:18 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

All tunnels - like all links, have both:
	- a largest message that can traverse the internal hops of that link
	- a largest message that can be reassembled at the egress of that link

These are generally referred to as “path MTU” and “EMTU_R” for IP.

> 1.       It could not be the case for tunneling that does not support fragmentation (L2TPv3, MPLS, VxLAN). Fragmentation buffer is just not needed in principle. Second restriction is not needed in…

That’s just the case where pathMTU == EMTU_R.

That means the EMTU_R is not needed *in practice*, but in principle it still exists.

> 2.       Reassembly buffer is needed for tunneling that does support fragmentation (NVO3, IPv6 GRE). But under no circumstances it should have any relationship to EMTU_R developed for TCP stack on…

EMTU_R is the IP reassembly buffer, as defined in RFC1122; it is not developed for TCP.

Vendors (and protocol designers) set EMTU_R > pathMTU so that fragmentation allows recursive tunneling (IPv4-in-IPv4, IPv6-in-IPv6). If they are equal, then you can never have direct recursive tunneling (you need a shim layer that creates a separate tunnel with fragmentation and reassembly).

For IPv4, min EMTU_R is 576 and min pathMTU is 68.
For IPv6, min EMTU_R is 1500 and min pathMTU is 1500.

These are different for a reason.

> That’s it – no second MTU restriction to pay attention to. It is the reason why this is not documented in any vendor’s documentation: no problem – no need to document.

RFC791 and RFC 2460 (and 8200) all define these.

> But you have assumed that vendors did always have fragmentation buffer with limited size related to EMTU_R. It is just not the case. Data Plane has no TCP stack to inherit TCP restrictions.

EMTU_R is the end-of-path reassembly. Vendors all implement this for packets terminating at routers. Vendors should (MUST) also implement this for tunnels that egress at the router; some do, some have an *implementation error* if they do not.

> Again: the current standards and implementations have only 1 MTU restriction per tunnel.

Any tunnel where pathMTU == EMTU_R cannot be used to tunnel over itself. That is a standards and implementation flaw, not a feature.

> It does permit to inform the host about oversized packets (by PTB).

PTB means only “I *cannot* deliver packets this big”, not “if you send me packets that are smaller, I won’t have to fragment them over tunnels. If you can fragment over tunnels, you CAN deliver packets that big and sending PTB would be incorrect.

> draft-ietf-intarea-tunnels proposes new solutions with 2 MTU restrictions per tunnel (small/restricted reassembly buffer size and MTU on the tunnel path) that pushes for fragmentation between these…

Draft-tunnels describes what already exists, in terms that allow us to have conversations about these properties.

It describes how current ICMP messages are insufficient to support your tunneling concerns. 

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 23 Mar 2021 15:47:46 +0000

Hi Joseph,
Please, show any except (or section number) from any RFC that push vendor to use 1500B buffer for reassembly in the data plane on the transit node? (or any other number on the transit node)
Please, show any evidence (or just claim if you could not disclose) that any vendor has 1500B (or less) for reassembly in the data plane (on transit node).
Please, show the evidence that the fragmentation problem exists.

All your discussions for EMTU_R are not relevant – it is for *host* reassembly buffer.
Data Plane on the transit node has not been regulated before – It was not needed. Vendors have chosen big enough numbers that did not affect anybody so far.
It was effectively 1 MTU restriction per 1 virtual interface, not 2.

I do not accept this:
That’s just the case where pathMTU == EMTU_R.
That means the EMTU_R is not needed *in practice*, but in principle it still exists.
If particular tunneling technology does not have a reassembly buffer (reject fragmentation in principle) – we could not talk about it like it exists.
If you would look to microcode – you would not find it. We could go very far with such logic.

Eduard
Sent: Tuesday, March 23, 2021 5:59 PM

On Mar 23, 2021, at 2:18 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Joe

---

From: Joseph Touch <[email-masked]>
Date: Tue, 23 Mar 2021 10:59:30 -0700

Hi, Eduard,

> On Mar 23, 2021, at 8:47 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

The Internet defines:
	- nodes that create or consume IP packets are hosts
	- nodes that relay IP packets are routers

RFC1122/1123 summarizes host requirements. RFC1812 summarizes router requirements. - both for IPv4. For IPv6, the former is largely in the IPv6 spec RFC8200 and RFC8504 and the latter in RFC8200, though there are some useful observations in RFC7084.

To your question, assuming you’re speaking of IPv6, RFC8200 requires nodes that consume IPv6 packets to support 1500B reassembly.
RFC791 requires that nodes that consume IPv4 packets support 576B reassembly.

> Please, show any evidence (or just claim if you could not disclose) that any vendor has 1500B (or less) for reassembly in the data plane (on transit node).

I neither know nor care. That’s a compliance issue, not a standards issue.

> Please, show the evidence that the fragmentation problem exists.

This is described in detail in:
	RFC1858
	RFC4459
	RFC4944
	RFC6946
	RFC6980
	RFC7588
	RFC8021
	RFC8900
	
As well as nearly any tunnel protocol description.

> All your discussions for EMTU_R are not relevant – it is for *host* reassembly buffer.

See above; you need to understand that the Internet defines HOST as any device that creates or consumes IP packets.

> Data Plane on the transit node has not been regulated before – It was not needed. Vendors have chosen big enough numbers that did not affect anybody so far.

See above; that’s not correct.

> If particular tunneling technology does not have a reassembly buffer (reject fragmentation in principle) – we could not talk about it like it exists.

I don’t see why you’re stuck on this issue. You don’t have to implement a separate reassembly buffer to have an implementation behave exactly as if there was one with the same size as the pathMTU. The effect is the same - no reassembly. 

> If you would look to microcode – you would not find it. We could go very far with such logic.

I don’t take my cues from others’ implementations.

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 23 Mar 2021 19:10:12 +0000

Hi Joseph,
I am not much interested to discuss IPv4 now. (despite that 2 MTUs for one interface is absent there too)
Let’s look at your reference to RFC 8200.

Section 4.5: unlike IPv4, fragmentation in IPv6 is performed only by source nodes, not by routers along a packet's delivery path
It means that all these discussions about fragmentation and reassembly are not related to transit nodes. It is for the “source and destination nodes”.
The better terminology is “transit node”, “destination node” – like it is in RFC 8200, not “host” or “router”.

You see – nobody is asking vendors to be compliant with any reassembly buffers in transit. Because it was assumed that would be not reassembly at transit.
Hence, vendors had the freedom to choose a much bigger number than 1500 when reassembly did happen in reality (despite IPv6 architecture decisions).

Please, show any evidence (or just claim if you could not disclose) that any vendor has 1500B (or less) for reassembly in the data plane (on transit node).
I neither know nor care. That’s a compliance issue, not a standards issue.
It is not a compliance issue, because there is no regulation/standard to comply with. Vendors had the freedom and solved the problem easily.

This is described in detail in:
            RFC1858
            RFC4459
            RFC4944
            RFC6946
            RFC6980
            RFC7588
            RFC8021
            RFC8900
I did not ask for a general discussion. Of course, fragmentation is a big topic with many publications.
I did ask for any evidence that there is 2 MTU per 1 virtual interface and fragmentation problem as the result of this (when packet would come in between of these MTUs).

I don’t see why you’re stuck on this issue.
Because you are trying to introduce additional fragmentation to the area where it was absent before. The root cause is the introduction of the second MTU per interface (that is in the reality the buffer size).
2 MTUs for one interface is the innovation. It does not exist in any standard or any real implementation. It is invented only in draft-ietf-intarea-tunnels.
It is not just new names and new classification. It new things that does not exist in the real world. Harmful, because of additional fragmentation introduced..

Eduard
Sent: Tuesday, March 23, 2021 9:00 PM

Hi, Eduard,

On Mar 23, 2021, at 8:47 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

The Internet defines:

I do not accept this:

Joe

---

From: Joseph Touch <[email-masked]>
Date: Tue, 23 Mar 2021 12:43:37 -0700

> On Mar 23, 2021, at 12:10 PM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

Agreed.

> The better terminology is “transit node”, “destination node” – like it is in RFC 8200, not “host” or “router”.

Please see section 2 of RFC8200 (color added by me):

2 <https://tools.ietf.org/html/rfc8200#section-2>.  Terminology

   node         a device that implements IPv6.

   router       a node that forwards IPv6 packets not explicitly
                addressed to itself.  (See Note below.)

   host         any node that is not a router.  (See Note below.)

The term “transit node” does no appear in RFC8200.

The terms “source node” and “destination node” are used in RFC8200 but not defined in Sec 2. They are clearly hosts that originate IPv6 packets and hosts that consume IPv6 packets, respectively.

In an IPv6 tunnel, the tunnel ingress emits new packets with IP headers it adds using its IP address. That makes it a source node. Same for how the egress consumes those packets. 

From the perspective of the tunnel path, the ingress and egress are hosts and intermediate hop relays are routers.

From the perspective of the overall path, the tunnel is a link, either host/host, host/router, router/host, or router/router. A tunnel is not itself a router, however.

> You see – nobody is asking vendors to be compliant with any reassembly buffers in transit. Because it was assumed that would be not reassembly at transit.

Reassembly happens at tunnel egresses whether you want it to or not. 

> Hence, vendors had the freedom to choose a much bigger number than 1500 when reassembly did happen in reality (despite IPv6 architecture decisions).

1500 is the IPv5 minimum EMTU_R; vendors can always implement larger reassembly when they choose to.

> Please, show any evidence (or just claim if you could not disclose) that any vendor has 1500B (or less) for reassembly in the data plane (on transit node).

Here’s how to do it:
	- set interfaces to use 1280B packets
	- setup an IPv6 tunnel
	- send a 1280B packet through that tunnel

If you don’t implement reassembly, it won’t work. But it does. Everywhere.

> It is not a compliance issue, because there is no regulation/standard to comply with. Vendors had the freedom and solved the problem easily.

RFC8200 is the standard. Tunnel ingresses and egresses create and consume packets, so they act as hosts. I don’t care if they’re implemented on routers; routers implement lots of things as hosts (see e.g., RFC4201, Sec 3.1:

   ...A compliant host
   implementation MUST support (a) and (c) and a compliant security
   gateway must support all three of these forms of connectivity, since
   under certain circumstances a security gateway acts as a host

> I did not ask for a general discussion. Of course, fragmentation is a big topic with many publications.

You asked for *specific examples* of what vendors do. Those RFCs provide them.

> Because you are trying to introduce additional fragmentation to the area where it was absent before. The root cause is the introduction of the second MTU per interface (that is in the reality the b…

I have not introduced anything; I am describing an existing requirement of any device that consumes IP packets (i.e., acts as an IP destination). When it does so, it is a host. Tunnel egresses do that.

> It is not just new names and new classification. It new things that does not exist in the real world. Harmful, because of additional fragmentation introduced..

Draft-tunnels has been discussed and reviewed by int-area for over a decade. Nobody else has agreed with your assertions.

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 23 Mar 2021 20:52:29 +0000

Hi Joseph,

Currently, vendors have chosen some undisclosed big numbers for the reassembly buffer on the tunnel interface
Or no buffer at all for tunnels that do not support reassembly.
That does not create any additional restriction for MTU.
Nobody did believe (IPv4 or IPv6 – does not matter) that buffer requirements for end nodes are applicable for transit nodes.
Your liberty to apply the requirements and terminology of one to the other is not a good idea.
draft-ietf-intarea-tunnels propose to decrease reassembly buffer to “typical host EMTU_R (1500B) minus tunnel outer headers overhead” that would cause additional fragmentation.

As the compromise:
Could you change the default for “draft-ietf-intarea-tunnels Tunnel MTU” (reassembly buffer) to 9k? (to reflect reality)
I would still be not happy in the mail to any alias about calling parameters of transit node buffer by the terminology of end node buffer.
But if you would not create additional fragmentation – I would not have any complains in my draft in regards to draft-ietf-intarea-tunnels.
It could be the resolution.

Well, probably it is not a good compromise, because you have the logic through all your document. 9k (reality) would protrude out of your logic.
The logic itself is good. It is broken because the most basic assumption is wrong (before you did apply any logic).

The Data Plane on transit nodes should not behave
as the Control Plane on transit nodes or Transport Layer on end nodes!
It was the wrong assumption initially. Buffers should be different. Names should be different. Unification here is not possible.
It would be rejected by vendors anyway because reassembly is expensive, the one who would increase it – would get a competitive disadvantage.
It is easy to translate additional reassembly to $$ losses.

Eduard
Sent: Tuesday, March 23, 2021 10:44 PM

On Mar 23, 2021, at 12:10 PM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Agreed.

2<https://tools.ietf.org/html/rfc8200#section-2>.  Terminology

Here’s how to do it:

   ...A compliant host

Joe

---

From: Joseph Touch <[email-masked]>
Date: Tue, 23 Mar 2021 14:10:03 -0700

Hi, Eduard,

> On Mar 23, 2021, at 1:52 PM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,
>  

Actually, it does. 

> Nobody did believe (IPv4 or IPv6 – does not matter) that buffer requirements for end nodes are applicable for transit nodes.

Your assumptions are not correct; this is widely understood by the vendors I have spoken with, some of whom helped write the specs on this issue I cited.

> draft-ietf-intarea-tunnels propose to decrease reassembly buffer to “typical host EMTU_R (1500B) minus tunnel outer headers overhead” that would cause additional fragmentation.

I’m decreasing the EMTU_R of the tunnel egress by the encapsulation header - that’s just how the effective tunnel EMTU_R is computed. I don’t WANT it to start at 1500 - that’s the value we start with unless we have more information about a specific implementation (it’s the minimum required). If a vendor knows it is larger, then use that, of course.

> Could you change the default for “draft-ietf-intarea-tunnels Tunnel MTU” (reassembly buffer) to 9k? (to reflect reality)

Draft-tunnels does not set a reassembly buffer size. It *CITES* values defined by the encapsulation protocols discussed (IPv4, IPv6 in particular).

If you want 9K, you should use ATM or SONET tunnels (that already have that size) or submit a request to update IPv6 through RFC8200.

> I would still be not happy in the mail to any alias about calling parameters of transit node buffer by the terminology of end node buffer.

Tunnels are what they are. 

> It could be the resolution.

Draft-tunnels does not WANT to create additional fragmentation - it’s describing what already happens.

Now, I realize you want PTB inside the tunnel to be relayed outside the tunnel. I have already explained why that is incorrect. Beyond being simply wrong, it also would effectively break current tunnels because ICMPs are often blocked.

So instead of fragmenting, you’d end up with a black hole. You need to address that FIRST.

> Well, probably it is not a good compromise, because you have the logic through all your document. 9k (reality) would protrude out of your logic.

Nothing in my doc prevents endpoints from using their ACTUAL EMTU_R. I am just saying we cannot assume anything larger than 1500 without knowing otherwise. We also currently have no protocol to discover this automatically. But that’s not saying we should not have such a protocol. We can and should. It would be part of the tunnel configuration.

> It was the wrong assumption initially. Buffers should be different. Names should be different. Unification here is not possible.

You are simply and completely incorrect.

I don’t see a point in further trying to explain why. The approach in draft-tunnels works. The approach has been shown to support tunneling over 15 levels deep (tunnels in tunnels in tunnels…) - including using trace route, etc.

Joe

> On Mar 23, 2021, at 12:10 PM, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Joseph,
>  <>2 <https://tools.ietf.org/html/rfc8200#section-2>.  Terminology
> 
>  
>

---

From: Mark Smith <[email-masked]>
Date: Wed, 24 Mar 2021 08:27:13 +1100

On Wed, 24 Mar 2021, 07:53 Vasilenko Eduard, <[email-masked]>
wrote:

> Hi Joseph,
>
>
>
> Currently, vendors have chosen some undisclosed big numbers for the
> reassembly buffer on the tunnel interface
>
> Nobody did believe (IPv4 or IPv6 – does not matter) that buffer
> requirements for end nodes are applicable for transit nodes.
>

I don't think you're not fully understanding the difference between hosts
and routers. I'm guessing you're thinking and imagining them as physical
devices. The definitions are actually functional (and the dictionary
definition of the word "device" is not a physical device.)

Outer tunnel packet header DAs are obviously the tunnel end-point IP
addresses.

Which of these definitions from RFC 8200 describes what type of IPv6 node a
tunnel end point is?

" router       a node that forwards IPv6 packets not explicitly
                addressed to itself.

   host         any node that is not a router."

Tunnel packets *are* explicitly addressed to a node, so tunnel endpoints
are performing host functions on packets. If those host functions being
performed on tunnel packets at the tunnel endpoints require buffers, then
buffers need to be available.

Internally, a router as a device (your "transit node") is performing both
host and routing functions. The forwarding plane component is:

"a node that forwards IPv6 packets not explicitly addressed to itself" - a
router

The control plane, and other functions like tunnel endpoints, in a router
as a device, is:

"any node that is not a router" - a host

This is the same for IPv4:

RFC791,

"The internet protocol provides for transmitting blocks of data called
datagrams from sources to destinations, where sources and destinations are
*hosts* identified by  fixed length addresses."

(And as another example of device verses function. If you buy a router as a
device from a router vendor (rack mount, multiple interfaces, limited
LCD/LED display, console port), and then use it as a BGP Route Reflector,
such that it never forwards packets because it is never in a forwarding
path, is it still a "router"? It might look like one, but it is only
performing host packet processing of the BGP protocol.)

Your liberty to apply the requirements and terminology of one to the other
> is not a good idea.
>
> draft-ietf-intarea-tunnels propose to decrease reassembly buffer to “typical
> host EMTU_R (1500B) minus tunnel outer headers overhead” that would cause
> additional fragmentation.
>
>
>
> As the compromise:
>
> Could you change the default for “draft-ietf-intarea-tunnels Tunnel MTU”
> (reassembly buffer) to 9k? (to reflect reality)
>
> I would still be not happy in the mail to any alias about calling
> parameters of transit node buffer by the terminology of end node buffer.
>
> But if you would not create additional fragmentation – I would not have
> any complains in my draft in regards to draft-ietf-intarea-tunnels.
>
> Well, probably it is not a good compromise, because you have the logic
> through all your document. 9k (reality) would protrude out of your logic.
>
> The logic itself is good. It is broken because the most basic assumption
> is wrong (before you did apply any logic).
>
>
>
> *The Data Plane on transit nodes should not behave*
>
> *as the Control Plane on transit nodes or Transport Layer on end nodes!*
>
> It was the wrong assumption initially. Buffers should be different. Names
> should be different. Unification here is not possible.
>
> It would be rejected by vendors anyway because reassembly is expensive,
> the one who would increase it – would get a competitive disadvantage.
>
> *From:* Joseph Touch [mailto:[email-masked]]
> *Sent:* Tuesday, March 23, 2021 10:44 PM
> *To:* Vasilenko Eduard <[email-masked]>
> *Cc:* [email-masked]; int-area <[email-masked]>
> *Subject:* Re: Proxy function for PTB messages on the tunnel end
>
>
>
>
>
>
>
> It means that all these discussions about fragmentation and reassembly are
> not related to transit nodes. It is for the “source and destination nodes”.
>
>
>
> Agreed.
>
>
>    *router*       a node that forwards IPv6 packets not explicitly
>
>    *host*         any node that is not a router.  (See Note below.)
>
>
>
> The terms “source node” and “destination node” are used in RFC8200 but not
> defined in Sec 2. They are clearly hosts that originate IPv6 packets and
> hosts that consume IPv6 packets, respectively.
>
>
>
> In an IPv6 tunnel, the tunnel ingress emits new packets with IP headers it
> adds using its IP address. That makes it a source node. Same for how the
> egress consumes those packets.
>
>
>
> From the perspective of the overall path, the tunnel is a link, either
> host/host, host/router, router/host, or router/router. A tunnel is not
> itself a router, however.
>
>
>
>
>
> You see – nobody is asking vendors to be compliant with any reassembly
> buffers in transit. Because it was assumed that would be not reassembly at
> transit.
>
>
>
> Hence, vendors had the freedom to choose a much bigger number than 1500
> when reassembly did happen in reality (despite IPv6 architecture decisions).
>
>
>
> Please, show any evidence (or just claim if you could not disclose) that
> any vendor has 1500B (or less) for reassembly in the data plane (on transit
> node).
>
>
>
> Here’s how to do it:
>
> It is not a compliance issue, because there is no regulation/standard to
> comply with. Vendors had the freedom and solved the problem easily.
>
>
>
> RFC8200 is the standard. Tunnel ingresses and egresses create and consume
> packets, so they act as hosts. I don’t care if they’re implemented on
> routers; routers implement lots of things as hosts (see e.g., RFC4201, Sec
> 3.1:
>
>
>
>    ...A compliant host
>
>    under certain circumstances a security gateway *acts as a host*
>
>
>
> I did ask for any evidence that there is 2 MTU per 1 virtual interface and
> fragmentation problem as the result of this (when packet would come in
> between of these MTUs).
>
>
>
> Because you are trying to introduce additional fragmentation to the area
> where it was absent before. The root cause is the introduction of the
> second MTU per interface (that is in the reality the buffer size).
>
>
>
> I have not introduced anything; I am describing an existing requirement of
> any device that consumes IP packets (i.e., acts as an IP destination). When
> it does so, it is a host. Tunnel egresses do that.
>
>
>
> 2 MTUs for one interface is the innovation. It does not exist in any
> standard or any real implementation. It is invented only in
> draft-ietf-intarea-tunnels.
>
> It is not just new names and new classification. It new things that does
> not exist in the real world. Harmful, because of additional fragmentation
> introduced..
>
>
>
> Draft-tunnels has been discussed and reviewed by int-area for over a
> decade. Nobody else has agreed with your assertions.
>
>
>
> Joe
> [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 24 Mar 2021 10:51:06 +1300

> The terms “source node” and “destination node” are used in RFC8200 but not defined in Sec 2. They are clearly hosts that originate IPv6 packets and hosts that consume IPv6 packets, respectively.

In fact "destination node" is more than that, since in some contexts it is not the final destination (that presumably uses the transport header and the payload) but an intermediate router in the case of source routing, including SRV6.

Agreed that RFC 8200 does not discuss tunnel end points, which are legitimately also on the path. So RFC 2473 is definitive until updated.

By the way, please avoid colour in IETF messages. Everything needs to be clear for text/plain readers.

Regards
    Brian

-----------------
> *From:*Joseph Touch [mailto:[email-masked]]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 24 Mar 2021 16:47:15 +1300

On 24-Mar-21 10:27, Mark Smith wrote:
> 
> 
> On Wed, 24 Mar 2021, 07:53 Vasilenko Eduard, <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Hi Joseph,____
> 
>     __ __
> 
>     Currently, vendors have chosen some undisclosed big numbers for the reassembly buffer on the tunnel interface____
> 
>     Or no buffer at all for tunnels that do not support reassembly.____
> 
>     That does not create any additional restriction for MTU.____
> 
> I don't think you're not fully understanding the difference between hosts and routers. I'm guessing you're thinking and imagining them as physical devices. The definitions are actually functional (and the dictionary definition of the word "device" is not a physical device.)
> 
> Outer tunnel packet header DAs are obviously the tunnel end-point IP addresses.
> 
> Which of these definitions from RFC 8200 describes what type of IPv6 node a tunnel end point is?
> 
> Tunnel packets *are* explicitly addressed to a node, so tunnel endpoints are performing host functions on packets. If those host functions being performed on tunnel packets at the tunnel endpoints require buffers, then buffers need to be available.

Yes, and tunnel entry points are hosts when they send an encapsulated packet. There's a subtlety though, which is that the device that contains the tunnel entry point on the downstream side of the packet flow is indeed a router on the upstream side. Conversely,
the device that contains the tunnel exit point is a host on the upstream side and a router on the downstream side.

Which is why the MTU for packets inside the tunnel is no business of anyone upstream or downstream of the tunnel.

   Brian

> 
> Internally, a router as a device (your "transit node") is performing both host and routing functions. The forwarding plane component is:
> 
> "a node that forwards IPv6 packets not explicitly addressed to itself" - a router
> 
> The control plane, and other functions like tunnel endpoints, in a router as a device, is:
> 
> "The internet protocol provides for transmitting blocks of data called datagrams from sources to destinations, where sources and destinations are *hosts* identified by  fixed length addresses."
> 
> (And as another example of device verses function. If you buy a router as a device from a router vendor (rack mount, multiple interfaces, limited LCD/LED display, console port), and then use it as a BGP Route Reflector, such that it never forwards packets because it is never in a forwarding path, is it still a "router"? It might look like one, but it is only performing host packet processing of the BGP protocol.)
> 
> 
>     ____
> 
>     Your liberty to apply the requirements and terminology of one to the other is not a good idea.____
> 
>     draft-ietf-intarea-tunnels propose to decrease reassembly buffer to “typical host EMTU_R (1500B) minus tunnel outer headers overhead” that would cause additional fragmentation.____
> 
>     __ __
> 
>     As the compromise:____
> 
>     Could you change the default for “draft-ietf-intarea-tunnels Tunnel MTU” (reassembly buffer) to 9k? (to reflect reality)____
> 
>     I would still be not happy in the mail to any alias about calling parameters of transit node buffer by the terminology of end node buffer.____
> 
>     But if you would not create additional fragmentation – I would not have any complains in my draft in regards to draft-ietf-intarea-tunnels. ____
> 
>     It could be the resolution.____
> 
>     __ __
> 
>     Well, probably it is not a good compromise, because you have the logic through all your document. 9k (reality) would protrude out of your logic.____
> 
>     The logic itself is good. It is broken because the most basic assumption is wrong (before you did apply any logic).____
> 
>     __ __
> 
>     *The Data Plane on transit nodes should not behave____*
> 
>     *as the Control Plane on transit nodes or Transport Layer on end nodes!____*
> 
>     It was the wrong assumption initially. Buffers should be different. Names should be different. Unification here is not possible.____
> 
>     It would be rejected by vendors anyway because reassembly is expensive, the one who would increase it – would get a competitive disadvantage.____
> 
>     It is easy to translate additional reassembly to $$ losses.____
> 
>     __ __
> 
>     Eduard____
> 
>     *From:*Joseph Touch [mailto:[email-masked] <mailto:[email-masked]>]
>     *To:* Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>
>     *Cc:* [email-masked] <mailto:[email-masked]>; int-area <[email-masked] <mailto:[email-masked]>>
>     *Subject:* Re: Proxy function for PTB messages on the tunnel end____
> 
>     __ __
> 
>     __ __
> 
> 
> 
>     ____
> 
>         On Mar 23, 2021, at 12:10 PM, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:____
> 
>         __ __
> 
>         Hi Joseph,____
> 
>         I am not much interested to discuss IPv4 now. (despite that 2 MTUs for one interface is absent there too)____
> 
>         Let’s look at your reference to RFC 8200.____
> 
>         Section 4.5: unlike IPv4, fragmentation in IPv6 is performed only by source nodes, not by routers along a packet's delivery path____
> 
>         It means that all these discussions about fragmentation and reassembly are not related to transit nodes. It is for the “source and destination nodes”.____
> 
>     __ __
> 
>     Agreed.
> 
> 
>     ____
> 
>         The better terminology is “transit node”, “destination node” – like it is in RFC 8200, not “host” or “router”.____
> 
>     __ __
> 
>     Please see section 2 of RFC8200 (color added by me):____
> 
>     __ __
> 
> 
>         2 <https://tools.ietf.org/html/rfc8200#section-2>.  Terminology____
> 
>     __ __
> 
>     __ __
> 
>        node         a device that implements IPv6.____
> 
>     __ __
> 
>        *router*       a node that forwards IPv6 packets not explicitly____
> 
>                     addressed to itself.  (See Note below.)____
> 
>     __ __
> 
>        *host*         any node that is not a router.  (See Note below.)____
> 
>     __ __
> 
>     The term “transit node” does no appear in RFC8200.____
> 
>     __ __
> 
>     The terms “source node” and “destination node” are used in RFC8200 but not defined in Sec 2. They are clearly hosts that originate IPv6 packets and hosts that consume IPv6 packets, respectively.____
> 
>     __ __
> 
>     In an IPv6 tunnel, the tunnel ingress emits new packets with IP headers it adds using its IP address. That makes it a source node. Same for how the egress consumes those packets. ____
> 
>     __ __
> 
>     From the perspective of the tunnel path, the ingress and egress are hosts and intermediate hop relays are routers.____
> 
>     __ __
> 
>     From the perspective of the overall path, the tunnel is a link, either host/host, host/router, router/host, or router/router. A tunnel is not itself a router, however.____
> 
>     __ __
> 
>     __ __
> 
>         You see – nobody is asking vendors to be compliant with any reassembly buffers in transit. Because it was assumed that would be not reassembly at transit.____
> 
>     __ __
> 
>     Reassembly happens at tunnel egresses whether you want it to or not. ____
> 
> 
> 
>     ____
> 
>         Hence, vendors had the freedom to choose a much bigger number than 1500 when reassembly did happen in reality (despite IPv6 architecture decisions).____
> 
>     __ __
> 
>     1500 is the IPv5 minimum EMTU_R; vendors can always implement larger reassembly when they choose to.____
> 
> 
> 
>     ____
> 
>         Please, show any evidence (or just claim if you could not disclose) that any vendor has 1500B (or less) for reassembly in the data plane (on transit node).____
> 
>     __ __
> 
>     Here’s how to do it:____
> 
>                 - set interfaces to use 1280B packets____
> 
>                 - setup an IPv6 tunnel____
> 
>                 - send a 1280B packet through that tunnel____
> 
>     __ __
> 
>     If you don’t implement reassembly, it won’t work. But it does. Everywhere.____
> 
>     __ __
> 
>         I neither know nor care. That’s a compliance issue, not a standards issue.____
> 
>         It is not a compliance issue, because there is no regulation/standard to comply with. Vendors had the freedom and solved the problem easily.____
> 
>     __ __
> 
>     RFC8200 is the standard. Tunnel ingresses and egresses create and consume packets, so they act as hosts. I don’t care if they’re implemented on routers; routers implement lots of things as hosts (see e.g., RFC4201, Sec 3.1:____
> 
>     __ __
> 
>        ...A compliant host____
> 
>        implementation MUST support (a) and (c) and a compliant security____
> 
>        gateway must support all three of these forms of connectivity, since____
> 
>        under certain circumstances a security gateway *acts as a host*____
> 
>     __ __
> 
>         This is described in detail in:____
> 
>                     RFC1858____
> 
>                     RFC4459____
> 
>                     RFC4944____
> 
>                     RFC6946____
> 
>                     RFC6980____
> 
>                     RFC7588____
> 
>                     RFC8021____
> 
>                     RFC8900____
> 
>         I did not ask for a general discussion. Of course, fragmentation is a big topic with many publications.____
> 
>     __ __
> 
>     You asked for *specific examples* of what vendors do. Those RFCs provide them.____
> 
> 
> 
>     ____
> 
>         I did ask for any evidence that there is 2 MTU per 1 virtual interface and fragmentation problem as the result of this (when packet would come in between of these MTUs).____
> 
>                                                   ____
> 
>         I don’t see why you’re stuck on this issue.____
> 
>         Because you are trying to introduce additional fragmentation to the area where it was absent before. The root cause is the introduction of the second MTU per interface (that is in the reality the buffer size).____
> 
>     __ __
> 
>     I have not introduced anything; I am describing an existing requirement of any device that consumes IP packets (i.e., acts as an IP destination). When it does so, it is a host. Tunnel egresses do that.____
> 
> 
> 
>     ____
> 
>         2 MTUs for one interface is the innovation. It does not exist in any standard or any real implementation. It is invented only in draft-ietf-intarea-tunnels.                            ____
> 
>         It is not just new names and new classification. It new things that does not exist in the real world. Harmful, because of additional fragmentation introduced..____
> 
>     __ __
> 
>     Draft-tunnels has been discussed and reviewed by int-area for over a decade. Nobody else has agreed with your assertions.____
> 
>     __ __
> 
>     Joe____
> 
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 24 Mar 2021 09:00:59 +0000

Hi all,
I could not stop the discussion at this point, because I was incomplete in the last message. I have missed to mention one important point.
The difference between how tunnel and end node operate is in 2 things:

1.       The tunnel has an unlimited buffer that is not counted as the restriction. It is assumed always as “big enough” by all vendors.

2.       The incoming packet is not checked against the buffer size. It is checked against the MTU of this virtual/tunnel interface.
As a consequence: PMTUD could correct interface MTU, then the next oversized packet would inform (by PTB) the real traffic source. PMTUD works.

All of the above is broken by draft-ietf-intarea-tunnels:

1.       Buffer is assumed small (1500 minus headers)

2.       The incoming packet is checked against the buffer size, not the MTU of the virtual interface

3.       No feedback is proposed between interface MTU and the buffer size
As consequence: PMTUD could not propagate through such interface (PTB feedback from inside the tunnel would not activate at some point the PTB feedback in the traffic source direction). PMTUD is effectively finished intentionally.
Hence, the need for additional fragmentation for the case when the packet is below buffer size, but above tunnel interface MTU.

The market has lost RFC 2473 functionality: it did propose a PTB proxy to react to the 1st PTB message from inside the tunnel.
But the market has at least a slow reaction: the second oversized packet would inform traffic source (1st would be used to correct MTU on the virtual interface).
I had the message in draft-vasilenko-v6ops-ipv6-oversized-analysis: let’s return to the original functionality proposed by RFC 2473.

draft-ietf-intarea-tunnels is the movement in opposite direction: it proposes to completely scrape PMTUD for the environment with tunnels.

All this story is a good example of when vendors did solve the problem very effective.
It is probably because the problem was simple and vendors were trying to be transparent – not to introduce additional limitations visible for others.
The fact that the problem was not over-standardized and not regulated was the important enabler.
But IETF has the intention to introduce additional limitations:

-          PMTUD would be permanently broken inside the tunnel

-          More traffic should be fragmented

-          It is not specified what to do for many tunnels that do not support fragmentation and do not have buffers at all. By new specification - it should be.

Eduard
Sent: Wednesday, March 24, 2021 12:10 AM

Hi, Eduard,

On Mar 23, 2021, at 1:52 PM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Actually, it does.

As the compromise:

Joe

Eduard

Hi Joseph,

Agreed.

Here’s how to do it:

   ...A compliant host

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 24 Mar 2021 09:41:25 +0000

Hi Mark,
I do not want to use “router” terminology here, because the router has the control plane where all requirements to IP stack (including EMTU_R) are fully applicable.
Moreover, some routers could have service card and could have even real “application” where EMTU_R is applicable too.
How buffer should look like is very much regulated for traffic source and destination.
A router has the data plane for transit traffic where nobody did believe that buffers for reassembly are regulated by any rules. It is just Boolean “Yes/No” in all tunneling RFCs: fragmentation & reassembly “supported/not_supported”. Nobody did explain how – up to a vendor.
Hence, “router” is not good terminology here.

The “Host” in this context is more usable terminology – it is always “the end”. But tunnel directly from the host is the corner case.
Yes, it could be. Then the draft-ietf-intarea-tunnels is right and applicable.
OK. Rename it to: “IP Tunnels in the Internet Architecture Originated from Hosts”.
Well, would be the question for other end of such tunnel that is probably terminated “in transit” (in the data plane of some “access concentrator”).
Hence, it would be not possible to make draft-ietf-intarea-tunnels valid just by decreasing the scope.
Eduard
From: Mark Smith [mailto:[email-masked]]
Sent: Wednesday, March 24, 2021 12:27 AM
To: Vasilenko Eduard <[email-masked]>
Cc: Joseph Touch <[email-masked]>; v6ops list <[email-masked]>; int-area <[email-masked]>
Subject: Re: [v6ops] Proxy function for PTB messages on the tunnel end

On Wed, 24 Mar 2021, 07:53 Vasilenko Eduard, <[email-masked]<mailto:[email-masked]>> wrote:
Hi Joseph,

RFC791,

As the compromise:

Eduard
From: Joseph Touch [mailto:[email-masked]<mailto:[email-masked]>]

Hi Joseph,

Agreed.

Here’s how to do it:

   ...A compliant host

Joe

---

From: Joseph Touch <[email-masked]>
Date: Wed, 24 Mar 2021 07:36:29 -0700

Hi, Eduard,

I’ll try to be brief because we’ve covered most of these issues before. We can have an extended discussion off-list if needed.

Joe

> On Mar 24, 2021, at 2:00 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi all,

Neither the number nor the size of buffers used for tunneling can be unlimited. “Assuming” otherwise is simply ignoring the issue, resulting in the tunnel being a black-hole.

> 2.       The incoming packet is not checked against the buffer size. It is checked against the MTU of this virtual/tunnel interface.

Tunnels all implement source fragmentation and egress reassembly for IPv6. It may not be readily visible to you, but it is there as a parameter of the virtual tunnel interface on the ingress and egress.

> As a consequence: PMTUD could correct interface MTU, then the next oversized packet would inform (by PTB) the real traffic source. PMTUD works.

I’ve cited numerous RFCs that focus on why ICMP PTB cannot be relied upon, i.e., that PMTUD does not work. You might assume it does work because when it fails it creates a black hole, where packets are simply dropped and no errors are received (because the ICMPs are filtered or simply not sent).

> 1.       Buffer is assumed small (1500 minus headers)

Draft-tunnels reports existing limits of IPv6 mandatory minimums. Individual implementations can be larger but there is no mechanism to discover some of those values (EMTU_R in particular, at the IP layer)

> 2.       The incoming packet is checked against the buffer size, not the MTU of the virtual interface

Incoming packets are checked at the receiver against receive buffer sizes. If that receiver is a router and the packet is relayed, that packet is checked against the MTU of the tunnel virtual interface during the process of transmission (after encapsulation); it is there that it is source fragmented.

Note: if that latter step does not occur, as you assert, then PMTUD would fail.

> 3.       No feedback is proposed between interface MTU and the buffer size

On this we agree.

> As consequence: PMTUD could not propagate through such interface (PTB feedback from inside the tunnel would not activate at some point the PTB feedback in the traffic source direction). PMTUD is ef…

That occurred nearly 20 years ago and was the motivation for RFC4821, again because ICMPs are filtered.

> Hence, the need for additional fragmentation for the case when the packet is below buffer size, but above tunnel interface MTU.

Correct.

> I had the message in draft-vasilenko-v6ops-ipv6-oversized-analysis: let’s return to the original functionality proposed by RFC 2473.

RFC2473 has numerous errors; draft-tunnels lists them in section 5.2. Here is an annotated version of that text:

   o  IPv6 tunnels [RFC2473 <https://tools.ietf.org/html/rfc2473>] -- IPv6 or IPv4 in IPv6
       o Treats tunnel MTU as tunnel path MTU, not tunnel egress MTU

Following RFC2473 prevents IPv6 over IPv6 when the tunnel is constrained to IPv6 minimum path MTU.

       o Decrements transiting packet hopcount (by 1)

As Brian noted, this is incorrect. That is because a tunnel is a link and Internet routers are supposed to decrement the hop count at routers, not over links. Following RFC2473 would prohibit a tunnel from host to host. It also thus affects tunnel mode IPsec.

       o Copies traffic class from tunnel link to tunnel transit header

That should be a choice, not an assumption; most vendors allow the choice to copy or not.

       o Ignores IPv4 DF=0 and fragments at that layer upon arrival

(Ignore; this is an error to be deleted in the next update)

       o Fails to retain soft ingress state based on inner ICMP messages
          affecting tunnel MTU

Following this rule causes PMTUD - which you favor - to break.

       o Tunnel ingress issues ICMPs

This is a violation of RFC792 and 8200; the ICMPs issued are that of routers, not links. If the ingress is at the source host, these ICMPs would come frfrom a device that is not a router.

       o Fragments IPv4 over IPv6 fragments only if IPv4 DF=0
          (misinterpreting the "can fragment the IPv4 packet" as
          permission to fragment at the IPv6 link header)

This causes IPv4 over IPv6 to fail when it doesn’t need to, simply to adapt to the tunnel path MTU rather than the tunnel EMTU_R.

Note: this is the behavior you’re advocating and it needlessly breaks things because it cannot optimize.  

> draft-ietf-intarea-tunnels is the movement in opposite direction: it proposes to completely scrape PMTUD for the environment with tunnels.

If you believe that is true, please cite the relevant text to support this claim.

Draft-tunnels just admits that PMTUD doesn’t work; it doesn’t propose deprecating it.

> -          PMTUD would be permanently broken inside the tunnel

Again, that is already true both inside and outside the tunnel.

> -          More traffic should be fragmented

Yes - more fragments rather than simply drop. That should be a win, not a problem.

> -          It is not specified what to do for many tunnels that do not support fragmentation and do not have buffers at all. By new specification - it should be.

Draft-tunnels is intended as BCP; it’s not a protocol spec.

But if you want to know what to do with tunnels that do not support fragmentation, it is stop using them. They’re broken.

Joe

---

From: Joseph Touch <[email-masked]>
Date: Wed, 24 Mar 2021 07:41:03 -0700

> On Wed, 24 Mar 2021, 07:53 Vasilenko Eduard, <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
...
> On Mar 23, 2021, at 8:47 PM, Brian E Carpenter <[email-masked]> wrote:
> 
> Yes, and tunnel entry points are hosts when they send an encapsulated packet.

Viewed from inside a tunnel, the tunnel endpoints are both hosts.

> There's a subtlety though, which is that the device that contains the tunnel entry point on the downstream side of the packet flow is indeed a router on the upstream side. Conversely,

I disagree; a tunnel is a link. That link can easily be host-host.

A host that is router-capable (many OSes have a flag to ‘forward IP’ or not) can have forwarding disabled (host-only mode) and use tunnels just fine.

> Which is why the MTU for packets inside the tunnel is no business of anyone upstream or downstream of the tunnel.

Agreed…

Though I would appreciate your thoughts on how to handle the errors in RFC2473 as noted in draft-tunnels.

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 24 Mar 2021 14:59:58 +0000

It would invalidate all tunneling implementations. It is not compatible with any one of them. PMTUD is killed. Revolution.

All complaints against RFC 2473 are minor (if right),
Except this one that is definitely wrong:

This is a violation of RFC792 and 8200; the ICMPs issued are that of routers, not links. If the ingress is at the source host, these ICMPs would come from a device that is not a router.
ICMP PTB is very important to deliver to the traffic source.

Ed/
Sent: Wednesday, March 24, 2021 5:36 PM

Hi, Eduard,

Joe

On Mar 24, 2021, at 2:00 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi all,

On this we agree.

Correct.

   o  IPv6 tunnels [RFC2473<https://tools.ietf.org/html/rfc2473>] -- IPv6 or IPv4 in IPv6

          affecting tunnel MTU

Joe

---

From: Joseph Touch <[email-masked]>
Date: Wed, 24 Mar 2021 12:00:43 -0700

Two points:

> On Mar 24, 2021, at 7:59 AM, Vasilenko Eduard <[email-masked]> wrote:
> 

PMTUD is effectively dead, so if you’re worried about it, you’re 20+ years too late - as per the RFCs I’ve already cited.

> ICMP PTB is very important to deliver to the traffic source.

I’m saying something very specific:
	- tunnels are links
	- links NEVER *genenerate* ICMPs
	- routers and hosts *generate* ICMPs
		based on what happens inside them, e.g,, to their processes and links

So the question is “under what conditions does a link cause a router/host to generate an ICMP?”

There should be no unsolicited ICMPs, i.e., routers/hosts NEVER generate ICMPs unless in reaction to a packet being sent or received.

PTB means “I cannot send the packet over this link”. Not path - link. There is no PTB for a path; the assumption is that one link of a path that fails will send the ICMPs back to the source.

For a tunnel, when can it NOT send a packet?
	- only when that packet is larger than the tunnel EMTU_R (i.e., egress received max, reassembled if reassembly is supported)

A packet that can be fragmented and traverse a tunnel is not too big. It’s “bigger than you might like” or “bigger than desired”, but there is no ICMP to indicate that sort of ‘soft’ (non failure) error.

So what should happen:
	- tunnels ingress should know and update (if changing) the tunnel EMTU_R value
	- routers/hosts should use EMTU_R as the tunnel MTU
		again, because the tunnel path MTU is a preference; the tunnel EMTU_R is the actual strict limit
	- routers/hosts sending packets over a tunnel generate ICMP PTBs as needed
		again, the router/host generates the message, not the tunnel ingress
		this happens when the router/host tries to send a packet over out that tunnel interface that is larger than the tunnel MTU

So this all works, as long as ICMPs are relayed.

Draft-tunnels does not deprecate this behavior. It describes it and explains why this is the correct behavior.

Tunnel ingresses that relay PTBs inside are broken; they fail in ways they do not need to. That is the true error.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 25 Mar 2021 08:51:01 +1300

On 25-Mar-21 03:41, Joseph Touch wrote:
>> On Wed, 24 Mar 2021, 07:53 Vasilenko Eduard, <[email-masked] <mailto:[email-masked]> <mailto:[email-masked]>> wrote:
> ...
>> On Mar 23, 2021, at 8:47 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>

Agreed. In that case, the prefixes reachable by the tunnel will show up in the host's route table, and the tunnel offers whatever MTU it can. In the general case the devices containing the TEPs will be routers and they will usually participate in a routing protocol for the prefixes reachable via the tunnel. 

> Though I would appreciate your thoughts on how to handle the errors in RFC2473 as noted in draft-tunnels.

As they say in the British Parliament, I require notice of that question. I need to re-read the draft.

   Brian

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 24 Mar 2021 20:04:30 +0000

Hi Joseph,
You have presented below (and in many other messages) a long list of policies (extensive usage of “SHOULD”, “NEVER”, “MUST”)
That are new – would change how current tunnels operate
And are not justified by any reasoning.
It is religion, not technology.

Why virtual link could not send ICMP PTB (like on a physical link)? Just because… it is “unsolicited”. But one moment – any other PTB is unsolicited too - It is an event.

You have not answered any of my questions – you continue to promote the solution from the draft-ietf-intarea-tunnels putting some excerpts in a different order.

PS: I am especially sorry that draft-ietf-intarea-tunnels would scrape the best tunneling RFC that we have for IPv6. RFC 2473 was really good.
Eduard
From: Joseph Touch [mailto:[email-masked]]
Sent: Wednesday, March 24, 2021 10:01 PM
To: Vasilenko Eduard <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>
Subject: Re: Proxy function for PTB messages on the tunnel end

Two points:

On Mar 24, 2021, at 7:59 AM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

So what should happen:

Joe

---

From: Joseph Touch <[email-masked]>
Date: Wed, 24 Mar 2021 13:50:18 -0700

Eduard,

> On Mar 24, 2021, at 1:04 PM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,

Some do, yes. To make them consistent and correct.

> Why virtual link could not send ICMP PTB (like on a physical link)? Just because… it is “unsolicited”. But one moment – any other PTB is unsolicited too - It is an event.

PTBs are only sent when a packet arrives at a router and the router decides to forward that packet to a link whose link MTU is too small.

If the link MTU changes, there would never be an ICMP PTB *until* a new packet is sent.

Go ahead - change the MTU of any interface (tunnel or not) of any router and tell me if you see a PTB being sent *ANYWHERE*.

Nothing happens until the next packet goes to that interface. I.e., PTBs are always in response to an attempt to send a packet - not merely to an interface change.

Joe

---

From: "Black, David" <[email-masked]>
Date: Wed, 24 Mar 2021 21:09:32 +0000

Hi Eduard,

>>            - links NEVER *generate* ICMPs
> Why virtual link could not send ICMP PTB (like on a physical link)?

The short answer (IMHO) is "yes, but the host or router generates the ICMP PTB."  There's a subtle distinction here that better supports the link-attached-to-host case, where an ICMP PTB may be counterproductive.

Starting with a router case, suppose that a large packet arrives at a router whose forwarding table determines that the next hop is a virtual link that encapsulates the packet to send in a tunnel.  The router checks the MTU for that virtual link (tunnel EMTU_R), determines that the packet is too large to send, and the router generates an ICMP PTB to report that.  This works the same way for a physical link that just sends the packet (MTU that is checked is that of the physical link) – in both cases the router generates an ICMP PTB based on link (interface) information before attempting to send the packet on that link.

The host case enables a host device driver or network stack to deal with this situation without forcing an ICMP PTB to be generated and parsed.  Starting with the physical link case – attempting to send a packet that's too large for the NIC to send generates an error that propagates back up the host network stack – forcing generation of an ICMP PTB in this case may be counterproductive.  Exactly where that error is generated may depend on how  the network stack and NIC are implemented – the error could originate from the NIC itself, the NIC device driver, or a higher layer of the network stack that checks the link MTU before the packet is handed off to the NIC device driver.  For a software encapsulation implementation of a virtual link, the MTU (tunnel EMTU_R) gets checked at or above the network stack layer that does the encapsulation.  If there's a software router embedded in the host (e.g., virtual switch with IP forwarding functionality), that router could generate an ICMP PTB based on the error or on directly checking the link MTU.

Thanks, --David

From: Int-area <[email-masked]> On Behalf Of Vasilenko Eduard
Sent: Wednesday, March 24, 2021 4:05 PM
To: Joseph Touch
Cc: [email-masked]; int-area
Subject: Re: [Int-area] Proxy function for PTB messages on the tunnel end

[EXTERNAL EMAIL]
Hi Joseph,

Two points:

So what should happen:

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 09:05:29 +0000

Hi Joseph,
It is exactly that I do not like most: complete eradicate PMTUD from any tunnel environment. Just prohibit it.
RFC 2473:
6.7<https://tools.ietf.org/html/rfc2473#section-6.7> IPv6 Tunnel MTU
The tunnel MTU is set dynamically to the Path MTU between the tunnel entry-point and the tunnel exit-point nodes, minus the size of the tunnel headers
…

The tunnel entry-point node performs Path MTU discovery on the path between the tunnel entry-point and exit-point nodes

Eduard
Sent: Wednesday, March 24, 2021 11:50 PM

Eduard,

On Mar 24, 2021, at 1:04 PM, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi Joseph,

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 09:51:53 +0000

Hi David,

1.       Any real tunneling implementation does check the incoming packet against virtual link MTU, not against any buffer. Because buffers are “big enough” for tunneling that does support fragmentation/reassembly or buffer does not exist at all for tunneling that does not support fragmentation.

2.       Why you are talking about “NIC”, “driver”, “EMTU_R”? All these abstractions do not exist in hardware that is doing tunneling for us.
It is the Data Plane ASIC, right? It deals with traffic flow (Verilog), not with control flow (RTC == Run to Completion, “C”).
It does not have LINUX on board. No one would be capable to emulate “host” on this ASIC, at least not to degree that this draft demands.
It was originally a very bad idea to unify tunnel end point with host – they are running on principally different hardware. The difference is much bigger than between RISC and CISC.

Eduard
From: Black, David [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 12:10 AM
To: Vasilenko Eduard <[email-masked]>; Joseph Touch <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>; Black, David <[email-masked]>
Subject: RE: Proxy function for PTB messages on the tunnel end

Hi Eduard,

Thanks, --David

From: Int-area <[email-masked]<mailto:[email-masked]>> On Behalf Of Vasilenko Eduard
Sent: Wednesday, March 24, 2021 4:05 PM
To: Joseph Touch
Cc: [email-masked]<mailto:[email-masked]>; int-area
Subject: Re: [Int-area] Proxy function for PTB messages on the tunnel end

[EXTERNAL EMAIL]
Hi Joseph,

Two points:

So what should happen:

Joe

---

From: "Black, David" <[email-masked]>
Date: Thu, 25 Mar 2021 16:46:25 +0000

Inline …

Thanks, --David

From: Vasilenko Eduard <[email-masked]>
Sent: Thursday, March 25, 2021 5:52 AM
To: Black, David; Joseph Touch
Cc: [email-masked]; int-area
Subject: RE: Proxy function for PTB messages on the tunnel end

[EXTERNAL EMAIL]
Hi David,

  1.  Any real tunneling implementation does check the incoming packet against virtual link MTU, not against any buffer. Because buffers are “big enough” for tunneling that does support fragmentation/reassembly or buffer does not exist at all for tunneling that does not support fragmentation.
[David>] We agree on this.  The check against the virtual link MTU (which is EMTU_R in the draft) is what causes generation of the ICMP PTB.

  1.  Why you are talking about “NIC”, “driver”, “EMTU_R”? All these abstractions do not exist in hardware that is doing tunneling for us.
[David>] That is entirely about "the link-attached-to-host case, where an ICMP PTB may be counterproductive" which also needs to be covered, in addition to the router case that you're focused on.

It is the Data Plane ASIC, right? It deals with traffic flow (Verilog), not with control flow (RTC == Run to Completion, “C”).
[David>] That would be in a router, actual hosts that don't have data plane ASICs also have to be covered.

It does not have LINUX on board. No one would be capable to emulate “host” on this ASIC, at least not to degree that this draft demands.
[David>] The draft specifies abstractions – implementations optimize across them.

It was originally a very bad idea to unify tunnel end point with host – they are running on principally different hardware. The difference is much bigger than between RISC and CISC.
[David>] I disagree - tunnel encap/decap software absolutely also runs on actual hosts.  Remote access VPN clients are an obvious example.

Eduard
From: Black, David [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 12:10 AM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Joseph Touch <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; int-area <[email-masked]<mailto:[email-masked]>>; Black, David <[email-masked]<mailto:[email-masked]>>
Subject: RE: Proxy function for PTB messages on the tunnel end

Hi Eduard,

Thanks, --David

[EXTERNAL EMAIL]
Hi Joseph,

Two points:

So what should happen:

Joe

---

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 17:12:13 +0000

Hi David,
I do not agree with the position:
“Let’s specify a standard for the software platform, the guys developing specialized ASICs should solve their problem themselves”.
Tunneling is done now much often in these specialized ASICs, not in PCs.
Eduard
From: Black, David [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 7:46 PM
To: Vasilenko Eduard <[email-masked]>; Joseph Touch <[email-masked]>
Cc: [email-masked]; int-area <[email-masked]>; Black, David <[email-masked]>
Subject: RE: Proxy function for PTB messages on the tunnel end

Inline …

Thanks, --David

From: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>

[EXTERNAL EMAIL]
Hi David,

Eduard

Hi Eduard,

Thanks, --David

[EXTERNAL EMAIL]
Hi Joseph,

Two points:

So what should happen:

Joe

---

From: Joseph Touch <[email-masked]>
Date: Thu, 25 Mar 2021 10:44:03 -0700

Eduard,

> On Mar 25, 2021, at 2:05 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Joseph,
>  <>6.7 <https://tools.ietf.org/html/rfc2473#section-6.7> IPv6 Tunnel MTU

If you obey this section of RFC2473, you cannot implement IPv6 over IPv6 if the real network path MTU is 1280B, because then the tunnel would be 1280-hdrs, which is necessarily smaller than required by IPv6.

That is why this is incorrect. The correct text should be:

6.7 <https://tools.ietf.org/html/rfc2473#section-6.7> IPv6 Tunnel MTU
The tunnel MTU is set dynamically to the egress reassembly MTU of the tunnel exit-point node, minus the size of the tunnel headers
…
The tunnel entry-point node performs discovery of this egress reassembly MTU where possible, or assumes the current IPv6 default of 1500-hdrs. 

Joe