Subject: draft-ietf-intarea-tunnels concerns

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 10:52:33 +0000

Hi Experts,
I have not received answers (after a long message thread) for me to understand:

1.       It is assumed by the draft that Data Plane in the transit router operates right now exactly like a host. Then Generalization is attempted for IP stack operation like on a host.

It is not the case. Moreover, it is not possible in principle because the hardware is ASIC managing traffic flow, but the host is CPU “running to completion” for control flow. The architecture of hardware is completely different.

2.       It is additional complexity: 2 MTUs for one virtual interface instead of the current 1 in all real data planes. 1st MTU is the buffer size - called “Tunnel MTU”. 2nd MTU is the old tunnel MTU- called “MAP”.
It looks extremely bad after the decision that 1st MTU (buffer size) is static till some miracle would explain to us how it would become dynamic in the future.

3.       The draft has deprecated PMTUD and introduced fragmentation instead of it. To be precise: for all bulk traffic that would happen between MTUs.
Moreover, It is not explained what to do for tunneling that does not want fragmentation now (currently prefer PMTUD). Should all tunnels support fragmentation from now on? (L2TPv3, VxLAN, MPLS, RFC 2473)

4.       If PMTUD is deprecated, then why it is still used for the 2nd interface MTU? If it is dead, then it is dead, right? Anyone could have the conclusion that the 2nd MTU is static too.

5.       The draft does break all tunneling specifications. Is everything should be changed in production? It is the cost. For what reason?

It does affect IPv6 too – I had stumbled upon this problem from that direction. RFC 2473 is the best tunneling spec that would be damaged severely.
Hence, 6man and v6ops on the copy.

I decided to leave it here for the people that may search for it in the future.
Eduard

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 25 Mar 2021 15:12:28 +0000

Eduard, I did a quick pass through the intarea discussions and I see points raised that
have been repeated many times since Y2K and even since much longer before that.
Thankfully, there is now a solution called the “OMNI Adaptation Layer (OAL)”:

https://datatracker.ietf.org/doc/draft-templin-6man-omni/

Some might say that it is the “second-coming of AAL5”, and it is true there are
many similarities. Like IP over ATM, the OMNI interface has an MTU/MRU visible
to the IP layer and set to 9180 bytes since that is the maximum size that can be
well protected by CRC32. Like AAL5, within the OMNI interface the OAL has a
“cell size” that determines the size of each fragment that will be produced by
the adaptation layer below IP.

Unlike AAL5 where the cell size is fixed at 48 octets, however, the OAL sets
both a minimum cell size (termed “minimum Maximum Payload Size (MPS)”)
and a possibly larger per-path MPS discovered using probes if necessary. The
minimum/path MPS is the maximum-sized RFC2473 fragment that the OAL
can sneak through the path and be assured it won’t be dropped (silently or
otherwise) due to a size restriction. Without any probing, the minimum MPS
that can be assumed is 576 minus encapsulation overhead (i.e., the IPv4
minimum EMTU_R). But unlike AAL5, the OAL can always strive to find a
larger per-path MPS.

There is more to be said about the OAL, but I will leave it at that for now.
We can talk about PTB “hard” and “soft” errors later if there is interest.

Fred

---

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 16:21:37 +0000

Hi Fred,
As I remember, you do not try to invalidate all other tunneling solutions.
draft-ietf-intarea-tunnels is doing exactly this.

IMHO: If some hardware achievement would not give us reassembly almost for free
Then the general solution should not assume reassembly because it is still very expensive for hardware at tunnel end points.
Overlay/Underlay is very popular our days – we would see more and more tunneling. SRv6 is the best example based on RFC 2473.

But it could be fine for your environment because overall performance would not go many Tbps per 1 gateway.
It is still possible to rely on something like Network Processor.
You would do some tradeoff between cost and flexibility. A little more cost for your environment should not be the problem.
Eduard
From: Templin (US), Fred L [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 6:12 PM
To: Vasilenko Eduard <[email-masked]>; int-area <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: RE: draft-ietf-intarea-tunnels concerns

Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 25 Mar 2021 16:33:34 +0000

Eduard, I am meaning to represent this as a general purpose solution. Performance
critical routers in the middle of the network will never be asked to reassemble – only
end systems or leaf network routers near the end systems will fragment, and only
end systems or routers near the end systems will need to reassemble.

You are right that OMNI/OAL are not meaning to invalidate all other tunneling
solutions, however they will improve the integrity of other tunneling solutions.
Remember that tunnels over IPv4 that set DF=0 and do not include an integrity
check are open to corruption. OMNI/OAL when used in the presence of those
other tunnel types closes the integrity gap.

Fred

---

Hi Fred,

But it could be fine for your environment because overall performance would not go many Tbps per 1 gateway.
It is still possible to rely on something like Network Processor.
You would do some tradeoff between cost and flexibility. A little more cost for your environment should not be the problem.
Eduard
From: Templin (US), Fred L [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 6:12 PM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; int-area <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: draft-ietf-intarea-tunnels concerns

Fred

---

From: Vasilenko Eduard <[email-masked]>
Date: Thu, 25 Mar 2021 17:00:55 +0000

Fred,
You could not considerably cut traffic by talking about core boxes.
The modern design (DC at any layers) may not have any core router for the country as big as 80M of population. I did careful calculations for many countries.
Because of Direct lambda from aggregation to the limited number of DC or peering points.
But you could say that only the downstream interface should do fragmentation/reassembly. Uplink could be cheaper.
Hence, /2 is a reasonable assumption.
It is still huge.

My personal estimation that FBB would go flat at 3Mbps, MBB at 1Mbps (5+ years) average per subs in the busy hour.
Let average household would be 2.6 persons (I know many countries, but I did never research for the world).
7.8B of MBB subscribers. 3B of FBB. Something like 17Pbps of real traffic.
/2 to see downlink only. But *4 for redundancy and reasonable port utilization.
Who would pay for the different hardware 34Ptbs?

The numbers above could be calculated more carefully. The model could be more accurate.
But No, massive fragmentation would never happen.
Eduard
From: Templin (US), Fred L [mailto:[email-masked]]
Sent: Thursday, March 25, 2021 7:34 PM
To: Vasilenko Eduard <[email-masked]>; int-area <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: RE: draft-ietf-intarea-tunnels concerns

Fred

---

Hi Fred,

Fred

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Thu, 25 Mar 2021 17:25:55 +0000

Eduard, at this point I think you would need to gain a better understanding of the
OMNI/OAL proposal by reading the draft to understand what is being proposed.
You may also need to look at the AERO draft to understand how the AERO/OMNI
segment routing functions. The drafts will not be changing again unless the IETF
directs me to do so:

https://datatracker.ietf.org/doc/draft-templin-6man-aero/
https://datatracker.ietf.org/doc/draft-templin-6man-omni/

As can be seen in the drafts, only end systems or near-end systems need to fragment
or reassemble, and iperf3 shows that that can be done at the O(10Gbps) level or even
higher which is plenty good enough for leaf networks. The core only needs to forward
OAL carrier packets with sizes determined by the MPS and as such are guaranteed not
to trip over an MTU restriction, and with line rate forwarding enabled.

Please have  a look and let us know what you think.

Fred

---

Fred,

Fred

---

Hi Fred,

Fred

---

From: Joseph Touch <[email-masked]>
Date: Thu, 25 Mar 2021 10:40:39 -0700

Hi, Eduard,

I’ve repeatedly addressed these, so in the same spirit of “putting it all in one place”, here are the answers.

Joe

> On Mar 25, 2021, at 3:52 AM, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi Experts,

Tunnel endpoints act like hosts. ASIC hardware can and does support fragmentation and reassembly at high speeds, and has *for decades* (all ATM hardware did this).

> 2.       It is additional complexity: 2 MTUs for one virtual interface instead of the current 1 in all real data planes. 1st MTU is the buffer size - called “Tunnel MTU”. 2nd MTU is the old tunnel…

They exist, whether you consider them complex or not. Sometimes they’re the same value (i.e., when no fragmentation is supported over the tunnel) and sometimes they’re fixed and known a-priori, but none of that changes that.

> It looks extremely bad after the decision that 1st MTU (buffer size) is static till some miracle would explain to us how it would become dynamic in the future.

Nobody has claimed that the tunnel pathMTU (MAP in draft-tunnels) or EMTU_R (true tunnel MTU) cannot change. 

> 3.       The draft has deprecated PMTUD and introduced fragmentation instead of it.

Draft-tunnels is intended as BCP. It has no power to deprecate.

Draft-tunnels does not imply that PMTUD should be used less frequently; it merely repeats the observation known for over 20 yrs that ICMPs are largely blocked and reliance on PMTUD is only asking to experience black-holes.

> To be precise: for all bulk traffic that would happen between MTUs.

Any tunnel that is used directly recursively (X over X, i.e., L2TPv2 over L2TPv3, etc.) must have an EMTU_R larger than its pathMTU. If it does not, it cannot support that use.

Those are factual observations, not requirements to specifications claims.

> 4.       If PMTUD is deprecated, then why it is still used for the 2nd interface MTU? If it is dead, then it is dead, right? Anyone could have the conclusion that the 2nd MTU is static too.

PLPMTUD does not use ICMPs but still relies on these two MTUs.

> 5.       The draft does break all tunneling specifications. Is everything should be changed in production? It is the cost. For what reason?

Draft-tunnels breaks nothing; it observes that some tunneling specifications *are already inherently broken*.

Seeing a broken window and reporting it does not mean that you broke that window.

> It does affect IPv6 too – I had stumbled upon this problem from that direction. RFC 2473 is the best tunneling spec that would be damaged severely.

RFC2473 has errors - as noted in Sec 5.2 of v10 of draft-tunnels.

I welcome discussion on those errors as well as how draft-tunnels should proceed. It is intended for BCP, which cannot update other docs - but it does not itself specify anything. But it could (should?) update all the standards noted in Sec 5.2. 

Can anyone suggest how best to do that?

> I decided to leave it here for the people that may search for it in the future.

Same here.

> Eduard
> _______________________________________________
> v6ops mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/v6ops <https://www.ietf.org/mailman/listinfo/v6ops>