Subject: Re: [Int-area] WG Adoption Call: IP Fragmentation Considered Fragile

From: Joe Touch <[email-masked]>
Date: Wed, 1 Aug 2018 07:11:18 -0700

We all understand that many current NAT devices and their deployments are not compatible with IP fragmentation (v4 or v6).

That leaves us with two options:
	1. change IP, but that leaves us with problems for which we have no solution (encrypted payloads, other DPI devices that look further in, etc.)
	2. change NATs and how they’re deployed (to require reassembly or its equivalent before processing, to not be deployed except where they can act as the host they proxy for)

Both cost money and will have an impact.

#2 involves changing less devices AND has the benefit that we know it will work.

I see no good reason to continue to try #1 in the meantime.

Joe

---

From: Ole Troan <[email-masked]>
Date: Wed, 1 Aug 2018 17:22:04 +0200

But only if you continue to ignore that there are other IPv4 sharing mechanisms than NAT. 

Ole

> On 1 Aug 2018, at 16:11, Joe Touch <[email-masked]> wrote:
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Aug 2018 07:38:00 -0700

I am not ignoring them; I’m claiming that they all have the same inherent deployment and implementation limitations.

Just because operators/vendors “want” to do otherwise does not make it possible.

Joe

> On Aug 1, 2018, at 8:22 AM, Ole Troan <[email-masked]> wrote:
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 2 Aug 2018 17:02:43 +0200 (CEST)

> Just because operators/vendors “want” to do otherwise does not make it 

I've been on hotel wifis that are behind 3 layers of NAT, PMTUD 
non-working, PMTU is like 1450, and the only thing saving the day is TCP 
MSS adjust, so the only thing that works is something over TCP or that 
happens to use small enough packets. I have been on other networks where 
basically only thing that works is 80/443 and some mail related ports. 
Complaining doesn't help, because peoples mobile phones work ok.

It's "possible", because it works well enough for what some people use it 
for. Very few complain, so there is no improvement.

So while you're technically and formally right, there is no enforcement 
and the only thing we can do is write requirements, tests, educate, but 
also educate application and protocol developers on what they might face 
in the real world. This is engineering, not physics. Real world is more 
important than map.

IP-fragmentation has always been fragile, and it's not improving. The 
Internet is growing, so this is not getting better. This is reality, even 
though we do not like it.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Aug 2018 08:19:07 -0700

> On Aug 2, 2018, at 8:02 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> I've been on hotel wifis that are behind 3 layers of NAT, PMTUD non-working, PMTU is like 1450, and the only thing saving the day is TCP MSS adjust, so the only thing that works is something over TCP or that happens to use small enough packets. I have been on other networks where basically only thing that works is 80/443 and some mail related ports. Complaining doesn't help, because peoples mobile phones work ok.
> 
> It's "possible", because it works well enough for what some people use it for. Very few complain, so there is no improvement.
> 
> So while you're technically and formally right, there is no enforcement and the only thing we can do is write requirements, tests, educate, but also educate application and protocol developers on what they might face in the real world. This is engineering, not physics. Real world is more important than map.
> 
> IP-fragmentation has always been fragile, and it's not improving. The Internet is growing, so this is not getting better. This is reality, even though we do not like it.

So you want us to redesign the Internet to run over port 443.

As you said, “this is reality, even if we don’t like it”.

The again, IP has fragmentation. That too is reality, even if we don’t like it.

Again, something broken needs fixing. You can chase the symptoms forever or you can deal with the cause. It’s simply not tenable to ‘fix’ the internet to accommodate broken devices.

Joe

---

From: Ole Troan <[email-masked]>
Date: Thu, 2 Aug 2018 17:38:45 +0200

Joe,

> Just because operators/vendors “want” to do otherwise does not make it possible.

There was IETF consensus behind those documents (A+P). 
In the _new_ IPv4 Internet architecture the IPv4 header looks like this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2  3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |.    0x45      |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live |      6|17     |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Source Port           |     Destination Port          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

If the the ascii art comes through.

In contrast to NAT the address and port fields are not rewritten. Only used for forwarding.
The source port and destination port fields are shared between L3 and L4. With an out-of-band provisioning protocol informing the end system which ports are available for applications.

Ole

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 2 Aug 2018 17:50:59 +0200 (CEST)

> So you want us to redesign the Internet to run over port 443.

Nope.

> The again, IP has fragmentation. That too is reality, even if we don’t like it.

IP have lots of things. Hop-by-hop-headers for instance. Really bad idea.

> Again, something broken needs fixing. You can chase the symptoms forever 
> or you can deal with the cause. It’s simply not tenable to ‘fix’ the 
> internet to accommodate broken devices.

The thing here is that you haven't proposed a realistic way to deal with 
the problem. We do not have any enforcement mechanism.

Applications need to work when faced with adverse conditions. They can 
work less well, that's fine, but they still need to work.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 2 Aug 2018 10:07:24 -0700

> IP have lots of things. Hop-by-hop-headers for instance. Really bad idea.
Mikael,

Definition of hop-by-hop options might have been flawed in that they
were required to be processed by every node in the path. But with that
restriction relaxed, this now is the only feasible mechanism that
provides inband host to network or network to host signaling. IMO,
this is far better idea than all the approaches that have being do ad
hoc DPI into transport layers or even transport payload. Fortunately
this is one area that might progress. QUIC seems to have enough
traction and encrypts header to render DPI ineffective. If the QUIC
application wants to tell something to the network it can do that by
HBH (this is a premise of FAST).

> The thing here is that you haven't proposed a realistic way to deal with the
> problem. We do not have any enforcement mechanism.
>
> Applications need to work when faced with adverse conditions. They can work
> less well, that's fine, but they still need to work.
>
This leads to driving everything down to only support the least common
denominator. Problem is that we can never move things forward if
everyone is bound to LCD.

Tom

> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Aug 2018 12:07:24 -0700

> everyone is bound to LCD.

Yup. As I said, we then need to reinvent the Internet over port 443. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2018 12:15:45 -0700

On 2018-08-02 08:38, Ole Troan wrote:

> Joe, 
> 
>> I am not ignoring them; I'm claiming that they all have the same inherent deployment and implementation limitations.
>> 
>> Just because operators/vendors "want" to do otherwise does not make it possible.
> 

You mean the *experimental* RFCs that describe an approach that doesn't
update RFC791? (i.e., RFC6364?) Or something else? 

> In the _new_ IPv4 Internet architecture the IPv4 header looks like this: 

A+P didn't update 791. There is no *new* IP header. 

The diagram above is a combination of IP - without options, notably -
and only two specific transports. It isn't an IPsec'd packet, a tunneled
packet, or any other transport. The Internet is not merely TCP and UDP
over IP with no options. 

> In contrast to NAT the address and port fields are not rewritten. Only used for forwarding.

And there may be limits to where that kind of approach can be deployed.
The jury is still out - this is experimental. 

Joe

---

From: Ole Troan <[email-masked]>
Date: Thu, 2 Aug 2018 21:33:06 +0200

Joe,

> You mean the *experimental* RFCs that describe an approach that doesn't update RFC791? (i.e., RFC6364?) Or something else?

The protocol specifications of A+P are all standards track.
RFC7596, RFC7597, RFC7599.

> The diagram above is a combination of IP - without options, notably - and only two specific transports. It isn't an IPsec'd packet, a tunneled packet, or any other transport. The Internet is not merely TCP and UDP over IP with no options.

For the public IPv4 Internet it is. (Sure there is some support for ICMP as well).

> And there may be limits to where that kind of approach can be deployed. The jury is still out - this is experimental.

There’s plenty of room for architectural purity in IPv6. Unfortunately there isn’t that luxury in IPv4 any more.

Ole

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2018 12:55:28 -0700

Thanks for the refs. Note that none of those update RFCs 791 or 1122, so
if frag breaks them, then it's their error. 

It also looks like (at first glance at least) these devices work only
when there isn't multipath between the back and front side. 

Joe

---

From: Ole Troan <[email-masked]>
Date: Thu, 2 Aug 2018 22:06:32 +0200

Joe,

> Thanks for the refs. Note that none of those update RFCs 791 or 1122, so if frag breaks them, then it's their error.

I wouldn’t be surprised if there were disagreements about that interpretation of “updates”.

> It also looks like (at first glance at least) these devices work only when there isn't multipath between the back and front side.

The A+P routers are stateless and do support multipath. Including traffic does not need to be symmetric.
That’s the main selling point for A+P, that you don’t need per flow state in the core of the network.

Cheers,
Ole

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Aug 2018 20:31:22 -0700

> On Aug 2, 2018, at 1:06 PM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 
> 

That’s not how it works, any more than there are disagreements over “standards track”.

Those docs are either compatible with existing specs, update those specs, or are in error. RFC791 takes precedence, having come earlier - until it is overridden by an update.

> That’s the main selling point for A+P, that you don’t need per flow state in the core of the network.

The +P part doesn’t seem like it’s compatible with fragmentation, though - yet it doesn’t update RFC791 to deprecate it throughout the Internet.

The only conclusion is that A+P should never be deployed in the presence of fragmentation - not that it should drop fragments, nor that we should consider deprecating fragmentation to address that flaw.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 3 Aug 2018 09:48:25 +0200 (CEST)

> everyone is bound to LCD.

I don't understand why people think this is what I am saying.

Car engines have "limp mode" when things go wrong. This is a restricted 
mode that gets you home, works much worse, but at least doesn't leave you 
stranded. I am saying applications should have the same.

I've kept saying "Networks must support ip fragmentation properly. 
Applications should still work somewhat when it doesn't".

Så relying on fragmentation and falling over when it doesn't work is not a 
good strategy to recommend to application developers.

---

From: Ole Troan <[email-masked]>
Date: Fri, 3 Aug 2018 11:54:02 +0200

Joe,

> The +P part doesn’t seem like it’s compatible with fragmentation, though - yet it doesn’t update RFC791 to deprecate it throughout the Internet.

It’s not incompatible with fragmentation. Just that there are some pitfalls. As explained in rfc7597. 
Fragmented packets do have a higher drop probability, and a higher probability of mis-reassembly on the end system with A+P.
Quite similar in behavior to a CGN. 

> The only conclusion is that A+P should never be deployed in the presence of fragmentation - not that it should drop fragments, nor that we should consider deprecating fragmentation to address that…

The problem A+P solves is to allow IPv4 to continue to grow outside it’s boundaries. The solution keeps session state close to the edge, so much more architecturally clean than CGN. 

If you can solve this problem in a better way please go ahead. 

Cheers 
Ole

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Aug 2018 07:18:19 -0700

> On Aug 3, 2018, at 2:54 AM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 

A flawed solution is not reason to break the rest of the Internet by deprecating fragmentation.

> ...If you can solve this problem in a better way please go ahead. 

I don’t have to, nor do I care to. But it isn’t productive to break another part of the Internet in the search for a solution to this problem.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 3 Aug 2018 09:00:19 -0700

On Fri, Aug 3, 2018 at 12:48 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> denominator. Problem is that we can never move things forward if everyone is
>> bound to LCD.
>
>
> Car engines have "limp mode" when things go wrong. This is a restricted mode
> that gets you home, works much worse, but at least doesn't leave you
You could say the same the thing about extension headers, SCTP and
DCCP, and even IPv6 itself since it still doesn't work everywhere. The
only protocols an application can _rely_ on working is TCP over plain
IPv4. That is current LCD. If the advice is that applications only use
protocols they can rely on, then Internet is stuck in time.

IMO, there should be a way for applications to use "alternative"
features and protocols with a fallback mechanism if necessary. For
instance, if the application had a priori knowledge that all nodes in
a path supported fragmentation, then there should be no issue with it
using fragmentation when sending on that path. Applying the car
analogy, if I look at a map and don't see any unpaved roads on the
route to my destination, then I can leave my four wheel drive all
terrain vehicle at home and take my Ferrari instead. I think that a
generalization of "Happy Eyeballs" might be a solution that discovers
and maps what features and protocols work on what paths.

Per the draft at hand, I think the advice to try to avoid
fragmentation is practical under the current circumstances. However, I
think that recommendation needs to be heavily qualified and scoped
appropriately. A general statement that applications shouldn't rely on
fragmentation cannot be interpreted as acceptance of non-conformant
implementation or as a free pass that middleboxes don't need to fix
there stuff.

Tom

---

From: Ole Troan <[email-masked]>
Date: Fri, 3 Aug 2018 19:21:38 +0200

> I don’t have to, nor do I care to. But it isn’t productive to break another part of the Internet in the search for a solution to this problem.

I don’t think that the runout of IPv4 addresses should come as a surprise to any one… nor that it has implications on the architecture of the Internet.
I think you are in the rough here…

Cheers,
Ole

---

From: Joe Touch <[email-masked]>
Date: Fri, 3 Aug 2018 10:26:08 -0700

> On Aug 3, 2018, at 10:21 AM, Ole Troan <[email-masked]> wrote:
> 
> I don’t think that the runout of IPv4 addresses should come as a surprise to any one…

It isn’t. 

> nor that it has implications on the architecture of the Internet.

It does already have implications on the implementation and deployment of devices. 

It has yet to be shown that the Internet architecture has to change, provided those existing limits are respected. 

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 4 Aug 2018 09:35:58 +0200 (CEST)

On Fri, 3 Aug 2018, Tom Herbert wrote:

> You could say the same the thing about extension headers, SCTP and DCCP, 
> and even IPv6 itself since it still doesn't work everywhere. The only 
> protocols an application can _rely_ on working is TCP over plain IPv4. 
> That is current LCD. If the advice is that applications only use 

I know plenty places where the only thing that works is TCP/80 and 
TCP/443. I know other places where the only thing usable is a SOCKS proxy. 
There are exactly zero protocols/ip version that an application can rely 
on. There are now operators which do not provide native IPv4 access, you 
have to use NAT64.

There is no such thing as a "sure thing".

> and maps what features and protocols work on what paths.

This is exactly what I am saying.

> Per the draft at hand, I think the advice to try to avoid fragmentation 
> is practical under the current circumstances. However, I think that 
> recommendation needs to be heavily qualified and scoped appropriately. A 
> general statement that applications shouldn't rely on fragmentation 
> cannot be interpreted as acceptance of non-conformant implementation or 
> as a free pass that middleboxes don't need to fix there stuff.

I would encourage everybody who wants the Internet to improve (and I 
support this), so get "someone" to fund work in freely available Internet 
access validation and testing. The best tool I know of is ICSI Netalyzr. I 
use this all the time to validate that what we come up with works for 
"everything". Yet, that tool is not complete and source code is not 
available. The CAIDA Spoofer project is also of interest.

So get someone with resources to allocate them to that work, that might 
actually improve things. Lots of people have no idea that their new 
fangled design of access solution doesn't work properly. I see this all 
the time "oh, we just turn on MSS adjust and everything works." "Err, what 
about fragmented IP packets and PMTUD?" "The what and the what???"

So this is not just about vendors and ISPs being evil and cheap, this is 
also largely of ignorance on all sides as well.

---

From: Wassim Haddad <[email-masked]>
Date: Mon, 13 Aug 2018 23:20:17 +0000

Hi,

The WG adoption call ended last Friday. From the discussion on the ML, we see support for adopting draft-bonica-intarea-frag-fragile-03 as a working group document.

Authors, please (re)-submit the draft as a WG document.

Thanks,
Juan & Wassim

> On Jul 24, 2018, at 12:42, Wassim Haddad <[email-masked]> wrote:
> 
> Dear all,
> 
> We would like to start a WG adoption call for draft-bonica-intarea-frag-fragile (“IP Fragmentation Considered Fragile”).
> 
> https://www.ietf.org/id/draft-bonica-intarea-frag-fragile-03.txt
> 
> 
> Please indicate your preferences on the mailling list. The deadline is August 10th.
> 
> 
> Thanks,
> 
> Juan & Wassim

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 25 Aug 2018 05:24:57 +0200

> I've kept saying "Networks must support ip fragmentation properly.

Why ? Wheren't you also saying that you've got (like probably many
else on this thread) all the experience that only TCP MSS gets you
working connectivity in many case (like hotels) ?

IMHO, we (network layer) should accept defeat on network layer 
fragmentation and agree that we should make it easier for the
transport layer to resolve the problem.

Aka: I would lvoe to see a new ICMPv4/ICMPv6 reply and/or PTB reply option
indicating "Fragmented Packets Not Permitted". Any network device which
for whatever reason does not like Fragemnts would simply drop
fragmented packets and send this as a reply. Allows then the
transport layer to automatically use packetization  (such as TCP MSS) 
to get packets through. 

Of course. Will take a decade to get ubiquitously deployed, but
neither IPv4 nor IPv6 will go away, only the problems with fragmentation
will become worse and work if we do not have an exit strategy like this.

If we don't try an exit strategy like this, we will just get what
Joe said, the complete segmentation of the Internet with more and
more L4 or even higher layer proxies.

Btw: +1 for adopting the doc as a WG item, but primarily because everything
before section 7 is on a way to become a good read of reality. Section
7 recommendations is only a faith based exercise (praying) as long as it tries to
get the job done primarily by appealing to application developers.

Cheers
    Toerless

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 25 Aug 2018 08:32:41 +0200 (CEST)

> working connectivity in many case (like hotels) ?

Correct. The reason for this is that whatever we design must be resilient 
to failure. Networks should work properly, but applications should 
handle it when they don't. Degraded performance is ok.

> transport layer to resolve the problem.

I want to keep the fragmentation requirement for the network.

> transport layer to automatically use packetization  (such as TCP MSS)

I am not opposed to this option being created, but you still need PLPMTUD. 
This option might trigger faster PLPMTUD, but it doesn't make the problem 
go away. If the application still keeps sending packets that needs to be 
fragmented, what should the stack do, just send an error to the 
application? Yes, this will mean we will fail faster, but apart from that?

---

From: Joel Jaeggli <[email-masked]>
Date: Sat, 25 Aug 2018 13:46:47 -0700

> transport layer to automatically use packetization  (such as TCP MSS) 

It's actually not that useful if it's an icmp message. because it's
going to fail in many cases where it has to be hashed to a destination.
just  like non-initial fragements do...

4821 gets you there with tcp.

> will become worse and work if we do not have an exit strategy like this.
It's not going to be ubiquitously deployed because it's not going to work.

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 09:09:54 -0700

> On Aug 24, 2018, at 8:24 PM, Toerless Eckert <[email-masked]> wrote:
> 
> more L4 or even higher layer proxies

FWIW, what I said was that *this exit strategy* would lead to the complete segmentation of the Internet and its consequences.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 10:08:07 -0700

> will become worse and work if we do not have an exit strategy like this.
Toeless,

I'm curious why you think the problems with fragmentation will become
worse. The draft and much of this thread has already highlighted the
problems with fragmentation that happen because of non-conformant
implementation. While there's a lot of legacy implementation that
might hard to fix completely, I don't think we've seen a good argument
that these problems are infeasible to fix in new deployments and
products. I think this draft is an opportunity not only highlight the
problems, but to suggest some practical fixes to improve the situation
as a way forward.

Tom

---

From: Christian Huitema <[email-masked]>
Date: Sun, 26 Aug 2018 10:31:53 -0700

It seems that the biggest obstacle to fragmentation are NAT and Firewall. They need the port numbers in order to find and enforce context. NAT might be going away with IPv6, maybe, but firewalls are not.

Have considered strategies that move the port number inside the IP header? For example, have an UDP replacement for IPv6 that does not have any port number in the new UDP header, and relies instead on unique IPv6 addresses per context?

-- Christian Huitema 

> On Aug 26, 2018, at 10:08 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 11:13:18 -0700

> Have considered strategies that move the port number inside the IP header? For example, have an UDP replacement for IPv6 that does not have any port number in the new UDP header, and relies instead…

Christian,

That's sort of along the lines for what Firewall and Service Tickets
(FAST) does. But instead of just putting a port number in the IP
header, FAST employs a generic ticket containing the state information
needed by the firewall. Tickets are encoded in Hop-by-Hop options, so
the firewall only needs to inspect the Hop-by-Hop options to do its
work eliminating the need for DPI at the middlebox (protocol compliant
with RFC8200). This works on any packet regardless of whether it's a
fragment (no reassembly at firewall is ever necessary), any
combination of extension headers, and any transport protocol even
those that don't have a concept of ports.

Tom

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 11:38:57 -0700

> On Aug 26, 2018, at 10:31 AM, Christian Huitema <[email-masked]> wrote:
> 

NATs already have what they need to do the proper job - they need to reassemble and defragment using unique IDs (or cache the first fragment when it arrives and use it as context for later - or earlier cached - fragments). There’s no rule that IP packets that are fragmented MUST have a transport header both visible (not encrypted) and immediately following the IP header. 

Firewalls are just delusions; the context they think they’re enforcing has no meaning except at the endpoints; it never did.

Using part of the IPv6 space for this solution would then break per-address network management (different UDP ports would use different IPv6 addresses, presumably).

The “disease" is that NATs don’t reassemble (or emulate it). It’s not useful to try to address the symptoms of that disease individually.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 12:58:00 -0700

Joe,

That is only a better solution, not a complete or robust one. For
reassembly to work all fragments of a packet must traverse the same
NAT device. There is no rule that IP packets going to the same
destination (fragments or not) ever MUST follow the same path. So in a
multi-homed environment this will eventually break someone. For IPv6,
this is also a clear violation of RFC8200 since intermediate nodes are
processing a non-HBH extension header in a packet not addressed to the
them. The only robust solution to NAT and its fragmentation problems,
as well as a bunch of other problems NAT brings, is to not use NAT!
(i.e. switch to IPv6)

Tom

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 14:12:20 -0700

> On Aug 26, 2018, at 12:58 PM, Tom Herbert <[email-masked]> wrote:
> 
> That is only a better solution, not a complete or robust one.

It is complete and robust...

> For
> destination (fragments or not) ever MUST follow the same path.

Correct, but there has to be a rule that all packets in a NAT’d flow go through the same NAT or multiple devices that emulate a single NAT.

> So in a
> them.

A NAT is not a router. A router is not allowed to modify the IP addresses or port numbers.

When a NAT does these changes, it is acting as a proxy endpoint in the public Internet; as such, it is *required* to do whatever is necessary to ensure that its behavior follows that of a host.

> The only robust solution to NAT and its fragmentation problems,

As I’ve mentioned, there are rules under which a NAT is a valid Internet device, but it is simply not just a router.

Joe

---

From: Ole Troan <[email-masked]>
Date: Sun, 26 Aug 2018 23:26:47 +0200

> On 26 Aug 2018, at 23:12, Joe Touch <[email-masked]> wrote:
> 

If there really was, can you point to where those rules are? Describing the behavior of the host stack and applications?

Ole

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 26 Aug 2018 23:27:03 +0200

> I want to keep the fragmentation requirement for the network.

Why ? Whats the biggest benefit with IPv6 ?

Took us decades to figure out that in-network
fragmentation (as mandaory in IPv4) is not a good thing, and
we eliminated it for IPv6. Why do we hang on to fragmentation 
from the host when tranport layers would be better doing it than the IP
layer ?

Cheers
    Toerless

> This option might trigger faster PLPMTUD, but it doesn't make the problem go
> away. If the application still keeps sending packets that needs to be
> fragmented, what should the stack do, just send an error to the application?
> Yes, this will mean we will fail faster, but apart from that?
> 
> -- 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 26 Aug 2018 23:29:13 +0200

> 4821 gets you there with tcp.

Its meant to support 4821 in good networks. Its fine to always have
higher layer solutions that spend a lot of effort to work
in the worst possible networks below them, but that should not mean
that do not try to make the network below work better. That's all
that ICMP means to do - on the premise that we should open a door
of allowing networks NOT to support fragmented packets.

Cheers
    Toerless

> > get the job done primarily by appealing to application developers.

pub   DSA 1024/B67F56B2 2003-08-11 Joel Jaeggli <[email-masked]>
> sub ELG-E 4096/29407F92 2003-08-11

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 26 Aug 2018 23:31:34 +0200

> FWIW, what I said was that *this exit strategy* would lead to the complete segmentation of the Internet and its consequences.

I can't rmember i saw a good explanation why on the thhread and neither in
the draft - aka: Why we still need fragementation to keep the internet
from falling apart". I like 4821 and think its the way to go also for other
transports.

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 26 Aug 2018 23:55:58 +0200

On Sun, Aug 26, 2018 at 11:38:57AM -0700, Joe Touch wrote:
> NATs already have what they need to do the proper job - they need to reassemble and defragment using unique IDs (or cache the first fragment when it arrives and use it as context for later - or earlier cached - fragments). There???s no rule that IP packets that are fragmented MUST have a transport header both visible (not encrypted) and immediately following the IP header. 

Reassmbly/refragment and MTU discovery puts NAT out of the realm of many
cost effective HW acceleration methods. Simple address rewrite does not.

> Firewalls are just delusions; [1]
> the context they think they???re enforcing has no meaning except at the endpoints; it never did. [2]

I completely agree with [2], but my conclusion is not [1], but
rathat its highly valuable and necessary.

The ability of firewalls to open 5-tuple bidirectional pinholes because
of trigger traffic from the inside is IMHO the most important feature
to keep Internet hosts protected. I wish host stacks would be built securely,
but after a few decdaces i have given up on that for most hosts. Which is
why its so irritating when host stack pundits continue telling network device
stack builders what they should and should not do.

Firewalls inspecting unencrypted higher layer message elements where a fairly
well working security model based on having a separate security administration
from the application administration. Now the applications promise to
provide all the security themselves, but they primarily just prohibit visibility
of what they do, so its a lot harder to figure out when they are insecure.

Would you ever put all type of in-home "iot" gear thats not a Windows/MacOS
system with a GUI you can control on the Internet without a firewall ?

Cheers
    Toerless

> The ???disease" is that NATs don???t reassemble (or emulate it). It???s not useful to try to address the symptoms of that disease individually.
> 
> Joe

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 27 Aug 2018 00:03:22 +0200

On Sun, Aug 26, 2018 at 11:26:47PM +0200, Ole Troan wrote:
> > As I???ve mentioned, there are rules under which a NAT is a valid Internet device, but it is simply not just a router.
> 

"A NAT is a transport layer circuit proxy whose network stack owns
outside adresses on the inside and inside addresses on the outside."

Something like that. Very likely not explicitly defined that way given
how BEHAVE just developed pragmatically whats necessary to make
things work and AFAIK wa prudent enough not to have the architectural
argument .

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 15:34:42 -0700

> On Aug 26, 2018, at 2:26 PM, Ole Troan <[email-masked]> wrote:
> 
> 
> 

The principles are described and explained here:

Touch, J: Middlebox Models Compatible with the Internet <>. USC/ISI (ISI-TR-711), 2016. (

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 15:37:21 -0700

> On Aug 26, 2018, at 2:27 PM, Toerless Eckert <[email-masked]> wrote:
> 

See draft-ietf-intarea-tunnels

As I’ve noted repeatedly, any layer for which there is a maximum size ultimately needs a way to fragment and reassemble at that layer, otherwise it ceases to be a complete participant in the network stack (i.e., if fragmentation/reaseembly happens at another layer, then that layer is effectively shut out of being the basis of a tunnel or protocol layer without that other layer.

For IP, the simple issue is that the requirement for IP over IP (for IPsec tunnels, at a minimum), requires it.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 15:41:18 -0700

> On Aug 26, 2018, at 2:31 PM, Toerless Eckert <[email-masked]> wrote:
> 

As I noted before in this thread and in other discussions on this topic, requiring support for fragmentation at another layer merely pushes all of what IP does (endpoint addressing, message multiplexing, etc.) to that other layer, at which point you’re back where you start - that layer then will end up needing to support fragmentation in places and ways you don’t want to support because of its cost.

For IPv4, the ONLY problem with fragmentation is the limited fragment ID space, and that can be mitigated as per RFC 6864.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 15:50:18 -0700

> On Aug 26, 2018, at 2:55 PM, Toerless Eckert <[email-masked]> wrote:
> 

And crumple zones and airbags get in the way of cars running fast and being inexpensive.

I.e., you’re right - doing NATs properly is more expensive than lying that you have a functioning NAT that doesn’t.

> rathat its highly valuable and necessary.

Good. Continue to run them and tell your customers that they actually stop email when they block ports 23 and 110, etc.

The rest of us then will tunnel one port over another (VPN) and walk right through that device (like we all dll all the time in hotels).

> to keep Internet hosts protected.

A firewall hsa to be close enough to the endpoint to act as its proxy; at that point, provided it does act as its proxy, then that sort of 5-tuple filtering works fine. You’re offloading work of the host elsewhere - but that firewall needs to act as a true host proxy, which includes reassembly, or it won’t work properly (nor can it ever).

> I wish host stacks would be built securely,

No argument there - but again, pushing the work of that host to another device MAKES THAT DEVICE A HOST.

Hosts receiving packets reassemble. Period. Or they won’t work. Period. And those that don’t, don’t work.

> from the application administration.

No better, FWIW, than would be managed software inside the end system. There’s no strict rule these need to be separate devices, but - as per above - they work fine when they act as they actually need to.

> Now the applications promise to

Without firewall functions somewhere? No - I agree. But I also wouldn’t put that firewall inside the network where it couldn’t see the fragments to reassemble - because it will never work properly.

I.e., I agree that “it hurts when we do that”, but not that we have to do it the wrong way (even though it’s cheaper).

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 15:53:46 -0700

> On Aug 26, 2018, at 3:03 PM, Toerless Eckert <[email-masked]> wrote:
> 
> outside adresses on the inside and inside addresses on the outside.”

My definitions:
NAT: viewed as a router or link to devices on the private side, but a host on the public side [RFC3022]

Proxy: viewed as different hosts on each side [Sh86]

Transparent proxy: viewed, together with the source, as a single host to the sink but invisible to the source on both sides; these are sometimes called “performance-enhancing proxies” (PEP) [RFC3135] 

> things work and AFAIK wa prudent enough not to have the architectural

IMO, the different types of middleboxes can be defined in terms of the combinations of behaviors of existing defined network architecture components (see above).

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 16:01:28 -0700

On Sun, Aug 26, 2018 at 2:12 PM, Joe Touch <[email-masked]> wrote:
> It seems that the biggest obstacle to fragmentation are NAT and Firewall.
> They need the port numbers in order to find and enforce context. NAT might
> be going away with IPv6, maybe, but firewalls are not.
>
> Have considered strategies that move the port number inside the IP header?
> For example, have an UDP replacement for IPv6 that does not have any port
> number in the new UDP header, and relies instead on unique IPv6 addresses
> per context?
>
>
> NATs already have what they need to do the proper job - they need to
> reassemble and defragment using unique IDs (or cache the first fragment when
> it arrives and use it as context for later - or earlier cached - fragments).
> There’s no rule that IP packets that are fragmented MUST have a transport
> header both visible (not encrypted) and immediately following the IP header.
>
> Firewalls are just delusions; the context they think they’re enforcing has
> no meaning except at the endpoints; it never did.
>
> Using part of the IPv6 space for this solution would then break per-address
> network management (different UDP ports would use different IPv6 addresses,
> presumably).
>
> The “disease" is that NATs don’t reassemble (or emulate it). It’s not useful
> to try to address the symptoms of that disease individually.
>
> Joe,
>
> Correct, but there has to be a rule that all packets in a NAT’d flow go
> through the same NAT or multiple devices that emulate a single NAT.
>
Again, there is no such rule in IP. Multiple devices could emulate a
NAT, but the overhead of doing this down to the granularity of
fragmented packets would be extraordinary. The way implementations
have handled dynamic transport state in the network is to assume
consistent per flow routing, or only one egress point.

> A NAT is not a router. A router is not allowed to modify the IP addresses or
Maybe so, but a NAT is not a host either. NAT is one example of
intermediate nodes attempting participate in transport level protocols
and thereby breaking the end-to-end model. But, unlike a host, there
is no guarantee that it will be given all the necessary information to
produce the same behavior as a host. A set of assumptions external to
the protocols are required to achieve something close to correctness.
Best way to fix that is eliminate the need for dynamic state.
Accordingly, I think a recommendation in this draft should recommend
use IPv6 without NAT in order to address the NAT-fragmentation
problem.

Tom

> When a NAT does these changes, it is acting as a proxy endpoint in the
> public Internet; as such, it is *required* to do whatever is necessary to
> ensure that its behavior follows that of a host.
>
> As I’ve mentioned, there are rules under which a NAT is a valid Internet
> device, but it is simply not just a router.
>
If it's not a router, then I don't believe it could  could a host or

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 16:16:39 -0700

> stack builders what they should and should not do.
When the host stack pundits are asking network device stack builders
to conform to the standard protocols then I believe that is
reasonable. If firewalls were standard and ubiquitous, and standards
were adhered to, then host stacks would have no problem. But alas
they're not, so we're forced to implement the host stack per the least
common denominator functionality of network devices.

> system with a GUI you can control on the Internet without a firewall ?
Conversely, do you allow your smartphone to connect to a network
before you've verified that a firewall is being run in the network,
what vendor provided it, and what the configured rules are?

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 27 Aug 2018 01:33:50 +0200

On Sun, Aug 26, 2018 at 03:50:18PM -0700, Joe Touch wrote:
> I.e., you???re right - doing NATs properly is more expensive than lying that you have a functioning NAT that doesn???t.

Sometimes it helps designing solutions against what can actually be
achieved instead of wishfull thinking. The way we operate
against expectations on network devices today just creates a lot
of half-baked solutions.

> The rest of us then will tunnel one port over another (VPN) and walk right through that device (like we all dll all the time in hotels).

You can always prove the existance of wishfull thinking by
assuming all type of stupid advertisements or misunderstanding of
achievable functionality. But that does not disprove the
existance of useful or necessary functions.

> A firewall hsa to be close enough to the endpoint to act as its proxy; at that point, provided it does act as its proxy, then that sort of 5-tuple filtering works fine. You???re offloading work of the host elsewhere - but that firewall needs to act as a true host proxy, which includes reassembly, or it won???t work properly (nor can it ever).

If a host stack should do fragementation (as IPv6 mandates now).
If we think fragmentation is only something that needs to happen
for tunneling within the network stack then maybe not so much.

If i wouldn't have to worry about such proxy forwarding plane capabilities,
i definitely would prefer models like SOCKS. If i have to think about them
it becomes certainly difficult to even model this well.

> No argument there - but again, pushing the work of that host to another device MAKES THAT DEVICE A HOST.

Can i re-apply your argument about fragmentation: 

You said something like IP can not be self-sufficient enough if
it wouldn't support fragmentation because then you would have
to rely too much on the higher layers.

Your claim of requiring NAT to be hosts is because we do not permit
IP to be self-sufficient for address translation between different
domains. Aka: yes, logically today, NAT need to go up to
transport layer, which is bad. See Christians suggestion.

> Hosts receiving packets reassemble. Period. Or they won???t work. Period. And those that don???t, don???t work.

Hosts must have transport layer. Transport layer can do PLMTUD/transport layer
segmentation. No need for hosts to do IP layer fragementation.

> No better, FWIW, than would be managed software inside the end system. There???s no strict rule these need to be separate devices, but - as per above - they work fine when they act as they actually need to.

Microsoft provides some good enterprise system management to
separate application security management from application management
itself, but i am pretty sure there is no chance in hell to expand
that model across all type of hosts in a standardized fashion.
Hence its certainly very viable to figure out what the best is we
can do with firewall and other seucurity techniques on
"proxy" devices. See also MUD and the like.

> Without firewall functions somewhere? No - I agree. But I also wouldn???t put that firewall inside the network where it couldn???t see the fragments to reassemble - because it will never work properly.

Which circles us back to me questioning the need for fragement at
the IP layer (as opposed transport layer) in hosts that MUST have transport stack. vs. some
other type of devices that do e.g. not have transport stacks but want
to do tunneling IP in IP tunneling.

> I.e., I agree that ???it hurts when we do that???, but not that we have to do it the wrong way (even though it???s cheaper).

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 27 Aug 2018 02:09:32 +0200

> common denominator functionality of network devices.

[RANT]
Sure. And now we've got internet highways full of speeding, black, armored,
window tinted and removed license plate SUV packets. And given how
the road authorities are seen as commerical competitors to the business
models of those attack SUV packet companies they even manage to bribe 
congress into thinking that the road authorities should simply get out
of the way. And whenever you open one of those SUV cars, it's
full of little "net neutrality" crybabies running lacrimal gland 
attacks against the voting public.
[/RANT]

Aka: Its a commercial issue and standards are built these days to prohibit others
to do what you want to exclusively do yourself. I am saying this not
to discount the good standard results we have, but primarily to explain
why we do not also get other good standards.

> what vendor provided it, and what the configured rules are?

When pacemaker companies do willfully reject to fix security attack
vectors against their devices for years, the IETF should really start
focussing more on what it can do to create more network security
and the right architectures for it.

There should be a lot of business for all those crappy embedded
endpoint vendors to outsource security in a trusted fashion to
someone who cares about it.

Toerless

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 17:10:00 -0700

> against expectations on network devices today just creates a lot

Agreed, but reassembly is clearly possible (hosts do it). The issue is cost.

We are not in the business of defending a vendor's idea of profit margin
WHEN it gets in the way of a required mechanism. I've described why it's
required; you've indicated that it's expensive. So?

> existance of useful or necessary functions.
A function whose basic existence defies our current standards?

You admitted that devices that NAT in the middle of the net wouldn't
work because of a requirement of IP routing. So why aren't you trying to
change IP routing to fix the path and not vary - if you want to defend
the existence of mid-net NATs, then you have to change that requirement too.

I'm describing the rules for working within the existing requirements.
Changing fragmentation alone will not fix what's wrong with NATs or
firewalls in those cases.

> If a host stack should do fragementation (as IPv6 mandates now).

It currently does.

> for tunneling within the network stack then maybe not so much.
Because you think tunneling happens somewhere else? Tunneling happens at
host - BY DEFINITION. A device that adds a header with addresses *IS A
HOST* on the side where it emits those packets.

> it becomes certainly difficult to even model this well.
When you find a complete model better than the Internet, propose it.
Until then...
> to rely too much on the higher layers.
Yes.
> domains. 
No. NATs are hosts because the emit packets with new headers with
addresses they own. That's the very definition of a host on the side
where those packets are emitted.

> Aka: yes, logically today, NAT need to go up to
His suggestion is to make IP the one header where everything happens -
but then we don't have layering flexibility.

> Hosts must have transport layer.
Sure, and a transport layer that has no associations is vacuous. A host
is something that emits packets with their own addresses; minimally,
hosts MUST terminate/originate IP (a host that doesn't
terminate/originate IP does nothing). A host that doesn't
terminate/originate UDP or TCP is simply a host with no
associations/connections - which remains valid.

>  Transport layer can do PLMTUD/transport layer
Please describe how to implement IPsec tunnel mode in that case. 

> that model across all type of hosts in a standardized fashion.
Hmm. But there's a model to outsource it to a separate vendor and it
still works?

Anything that works with two vendors can work with one.

> "proxy" devices. See also MUD and the like.
The "best we can do", yes. To lie about what we can do, no.

> to do tunneling IP in IP tunneling.
They're also called hosts.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 17:11:18 -0700

> common denominator functionality of network devices.
Seriously, we cannot be wasting time making new rules for devices that
don't follow rules. What's the point?

> what vendor provided it, and what the configured rules are?

Nope. That's why I run a firewall *on the device*.

Joe

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 27 Aug 2018 04:35:13 +0200

> required; you've indicated that it's expensive. So?

Cost that is too high translates into "not going to happen". Else
we'd all be commuting in helicopters.

> A function whose basic existence defies our current standards?

I thought we where discussing evolution of our standards. 

> the existence of mid-net NATs, then you have to change that requirement too.

I think we're jumping a bit across various cases. Not that they are not
interesting. My main point was that we should separarte out
fragementation as something useful purely in device types without
necessary a full highler layer transport stack (like routers doing
tunneling at IP layer), and host stacks that should rather do
fragementation at transport stack or higher.  And yes, that would enable
me to make NAT and firewalls (for the firewall functions i think make sense)
for host stack traffic something that does not require to bother about
fragmentation and could therefore be done easier at higher speed
and architecturally as something only in the network layer. 

> firewalls in those cases.

The draft in question argues to limit what future work should do
within the existing requirements, which is fine. I was merely
pointing out that we could move more into what i think would be 
a useful evolution if we also went beyond our current arch
and evolved it. 

It's not really as if IPv6 itself did do a good job in trying to
figure out what network devices can and can not do within sellable
costs. And we're continuing to suffer from it.

> HOST* on the side where it emits those packets.

Sure. But lets not get stuck on current terminology of "host". Lets just figure out
what we think are the best rules where to apply fragmentation and why.
I think fragmentation is best pushed up on the stack. Packetization
fragmentation in the "higher layer" is IMHO better than fragments in
the lower layer. Even if the higher layer is a network layer
protocol itself.  

> When you find a complete model better than the Internet, propose it.

HTTPs over DWDM with application layer proxies on every hop.
You didn't define how to measure "better" ;-)

The example of SOCKS should have shown that i wasn't trying to replace
the internet architecture, but rather seeing what could be added on the
edge that is both as (IMHO) as useful as SOCKS but more lightweight.

> where those packets are emitted.

The architecture misses good terms to better characterize better the
type of nodes sitting in betweenwhat users would perceive to be hosts
(HTTPs/TCP stack and the like) and pure routers.

> but then we don't have layering flexibility.

Please explain what you think you would loose ? 
I see only benefits of moving demux identifiers one layer down.

> Please describe how to implement IPsec tunnel mode in that case. 

See terminology discuss. In my text you question, i was referring
to host' as something that can effficiently run TCP/HTTPs stacks,
not as hosts per TCP/IPv6 architecture terms. My hosts' would use 
transport mode.

If you're talkin about network devices using IPsec tunnel mode,
i would equally just pass the effective MTU up to avoid
fragmentation.

If i could build a network device doing fast fragmentation with
IPsec tunnel mode, i would probably look for how to most
easily extend IPsec to do such packet layer fragementation so as
to not bother the undelying layer.

> Hmm. But there's a model to outsource it to a separate vendor and it

Sure, why not ?
Before Microsoft after 25 years finally got some useful firewall
into windows 10, most firewalls where outsourced to separate vendors.

> Anything that works with two vendors can work with one.

Like checks and balances, division of power and the like. Don't think so.

> The "best we can do", yes. To lie about what we can do, no.

IMHO, logically the firewall is part of endpoint OS, but its impossible
to standardize that well especially with all that embedded/IoT crowd.
So its outsourced into a common network comonent, and the question
is what a reasonable amount of information about the apps is
that the OS should share with the outsources firewall to do the job.
Not really a difficult concept, just something that the single vendors
of endpoint OS never wanted to make work because of business reasons.

Cheers
   Toerless

-- 
---
[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Sun, 26 Aug 2018 20:01:07 -0700

> On Aug 26, 2018, at 7:35 PM, Toerless Eckert <[email-masked]> wrote:
> 

IPv6.

IP options.

And (perhaps) any new proposed solution.

We have to aim at what network components *need* to do to participate. IP fragmentation is exactly that.

> I thought we where discussing evolution of our standards. 

What we have are:
	- a (current) standard that works, but isn’t implemented in some places to save money
	- a proposed partial solution that works for only some protocols, and - you guessed it - will still cost money that some will want to save

Making a solution that is partial (won’t support IPsec at least) isn’t evolution; it’s devolution.

> fragementation at transport stack or higher.  

That’s easy to say, but since any host might be an IPsec tunnel endpoint, you’re back where we are now - needing IP fragmentation support everywhere, ultimately.

My view is simple - if you fix what we KNOW is wrong with NATs and firewalls - in KNOWN ways - then not only don’t we have to solve this problem, we don’t have a lot of other problems either (e.g., lack of state when a flow takes a different path into an enterprise).

> And yes, that would enable

You’re optimizing a long-term impact solution for a short-term limitation. That’s a bad idea; protocols last a very long time.

> a useful evolution if we also went beyond our current arch

I am fine with encouraging the *search* for new solutions, as long as *in the meantime* we also call out firewalls and NATs for how they are already broken. Until IP fragmentation is deprecated, that has to be our position as a community. Otherwise, we will be constantly redefining our protocols to how others mis-implement them (yes, I know a few people who write drafts with this very goal in mind, sadly).

> costs. And we're continuing to suffer from it.

We are continuing to suffer from cheaply made devices that claim Internet compliance but aren’t. Frankly, until we have a compliance arm, that’s not likely to change. However, we CAN avoid reacting as a community to the least-common denominator of what is implemented for profit.

I.e., a bake-off would catch this and flag it, not bury it as “well, the market decides”. The market cannot be trusted to create a consistent set of capabilities that will support networking.

> Sure. But lets not get stuck on current terminology of "host”.

Stop. No. If you can’t use definitions and behaviors that are the underpinning of the Internet architecture, then there’s no point in anything further.

> Lets just figure out

(see above for “market decides”; your way won’t necessarily get us to a useful network - it may patch one issue, but could easily open three others).

> I think fragmentation is best pushed up on the stack.

Again, please tell me how to do that for IPsec tunnels - which, again, can start/end anywhere in the network.

> Packetization

OK, how do you fragment IP without fragmenting IP when it’s IP over IP?

> You didn't define how to measure "better" ;-)

Agreed, but that’s exactly where you’re headed when you say “kick the can down the road to the upper layer protocol”.

> edge that is both as (IMHO) as useful as SOCKS but more lightweight.

Anything you add will end up being in something that looks and acts like either a firewall or a NAT and you’ll be right back here again, saying some other upper layer protocol should fix things.

> (HTTPs/TCP stack and the like) and pure routers.

Provide those terms, define their behavior in a way that is consistent with a full set of protocols, and call us when you’re done. I’ve done so by defining NATs and proxies in terms of existing architectural components, showing an equivalence. the alternative is to design a new, complete architecture.

> not as hosts per TCP/IPv6 architecture terms. My hosts' would use 

Then your solution works only for those things - which shouldn’t be called hosts (that’s already defined in RFC1122).

> i would equally just pass the effective MTU up to avoid

That can’t work - by the time the traffic gets to you, you don’t have control over whether the source will adjust its size of not (it might even be encrypted). Yet you might have to put a 1280 byte packet inside a new IPv6 payload. How do you do that without fragmenting? The space for the headers has to come from somewhere.

> to not bother the undelying layer.

You can’t - the packets you receive could be IPv6 and the packet you’re putting them in is IPv6. There’s no way to do this without redefining IPsec tunnel mode.

> into windows 10, most firewalls where outsourced to separate vendors.

Anything that can be outsourced can be incorporated.

> Like checks and balances, division of power and the like. Don't think so.

If you WANT them implemented separately, you can do that - except that you need to make sure that the firewall and host act as a single endpoint. You can’t delegate part of what a host does to something that fundamentally isn’t responsible for acting like a host (i.e., reassembling).

> of endpoint OS never wanted to make work because of business reasons.

Again, that’s OK as an implementation choice, but with that choice comes responsibility - the firewall is responsible for acting as a host. Not *wanting* to do that doesn’t make it so, nor does it make it possible or useful for us to try to patch the Internet to enable.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Sun, 26 Aug 2018 20:19:41 -0700

> that the OS should share with the outsources firewall to do the job.

Toerless,

I'm not sure what "outsourced into a common network component" means.
I've done a lot of app and OS development and have NEVER once
"outsourced" security to the network. OSes and apps need to work
across all networks, in any possible environment, so having one
network provide a strict firewall, and in the next one no firewall
doesn't help really help things. Least common denominator for security
is no firewall, and that's what we assume in host development.
However, we do have to jump through a number of hoops to satisfy some
of the implicit requirements created by firewalls.

> of endpoint OS never wanted to make work because of business reasons.
Or perhaps they don't want to make it work since there is no standard
protocol for hosts to communicate characteristics of traffic with the
network. I think https://datatracker.ietf.org/doc/draft-herbert-fast/
could be that.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 27 Aug 2018 07:21:57 +0200

On Sun, Aug 26, 2018 at 08:19:41PM -0700, Tom Herbert wrote:
> "outsourced" security to the network.

And i worked in a company where for a good while, SOCKS was a key part
of the security concept. What do two random people experience data points
help here ? ;-)

> OSes and apps need to work

The main question is what architecture we want for firewalls. IMHO i
primarily need one where the firewall operator can be someone else
from whoever operates any type of potentially crappy endpoint
or endpoint app. If there are perfect security endpoint/e dpoint apps, thats
fine, but they are not the problem anyhow.

Assuming host development to be security wise good enough to connect
to the internet without an external firewall is quite risky for most
hosts that are not running the latest Windows/MacOS with good firewall
configs. 

> network. I think https://datatracker.ietf.org/doc/draft-herbert-fast/

subscriber and app-id are probably more important.

Cheers
    Toerless

---

From: Ole Troan <[email-masked]>
Date: Mon, 27 Aug 2018 08:55:00 +0200

Joe,

> Touch, J: Middlebox Models Compatible with the Internet. USC/ISI (ISI-TR-711), 2016. (
> 

I don’t want to dismiss this completely, but it hand waves over how applications are supposed to work in this new Internet architecture. 
You can define your way out of breaking end-to-end, but that doesn’t mean you can ignore all the issues of NAT traversal.

Cheers 
Ole

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Aug 2018 01:27:43 -0700

> On Aug 26, 2018, at 11:55 PM, Ole Troan <[email-masked]> wrote:
> 
> Joe,
> 
>>> 

You’ve missed the point - if the middleboxes describe behave as required, apps do not need to change. They work as they would in an Internet without those boxes.

Joe

---

From: Ole Troan <[email-masked]>
Date: Mon, 27 Aug 2018 10:52:27 +0200

Joe,

> On 27 Aug 2018, at 10:27, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Quite likely.
Do you have a document describing how my SIP application works?
Ore are you saying PCP, ICE, TURN etc is part of your architecture?

Cheers 
Ole

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Aug 2018 06:43:26 -0700

Ole,

On 2018-08-27 01:52, Ole Troan wrote:

> ... 
> Touch, J: Middlebox Models Compatible with the Internet [1]. USC/ISI (ISI-TR-711), 2016. ( 
> 
> I don't want to dismiss this completely, but it hand waves over how applications are supposed to work in this new Internet architecture.  
> You can define your way out of breaking end-to-end, but that doesn't mean you can ignore all the issues of NAT traversal.

You've missed the point - if the middleboxes describe behave as
required, apps do not need to change. They work as they would in an
Internet without those boxes. 
Quite likely. 
Do you have a document describing how my SIP application works? 
Ore are you saying PCP, ICE, TURN etc is part of your architecture? 

This document provides the needed context in which to interpret apps and
network services. 

If you understand that the endpoints of the Internet SIP application are
the NAT and the server, not the host on the NAT's private side, you
understand most of what you need to know. I.e., the SIP client is trying
to trigger behavior in the NAT, but not knowing the source IP or source
port unless discovered or told directly. That's why in-band addresses
don't work without additional mechanisms such as PCP, ICE, and TURN. 

I'm not saying you don't need those mechanisms - but rather that the
model I propose explains what is needed and why directly. I.e., as long
as your app acts like it actually runs on the NAT box, it'll work just
fine; to the extent that you don't want to do that, the onus is on you
to support moving it to the hidden client host on the private side. 

Joe 

Links:
------
[1] http://webmail.strayalpha.com/./#NOP

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 29 Aug 2018 00:09:15 +0200

Thanks, Joe

This has gotten pretty long. Let me sumarize my suggestions upfront:

For the draft itself, how about it also consideres recommendations not only
for IPv6 but IPv4. Such as simply also only do what we've accepted to be
feasible for IPv6. Like: do never rely on in-network fragmentation but
only use DF packets.

The remaining considerations are more generic, and i wonder if this draft
wants to have the guts to even mention them. Probably not. But IMHO
we will continue to be architecturally stuck in the hole we are if we do not
tackle them for poor middleboxes:

The IETF may think they are bad and not needed, but reality does need middleboxes
that function at linerate of only per-packet inspection and not at the lower
speed achievable with "virtual reassembly based inspection". Therefore we
need options to have in every fragment all the same context that middleboxes
reasonably should be able to inspect. 

Its IMHO a clean pragmatic solution to say that this additional context can
be in the higher-layer protocol and that layer willingly exposes it to middleboxes -
and encrypts everthing else. This view then requires that higher-layer
protocol to perform packet layer fragmentation. Like we can do with
TCP. Thats why that approach is IMHO a good recommendation to use. If
other transport protocols want to support the same degree of interaction
with middleboxes. Fine. If not, fine too.

Given how we do have TCP PLMTUD, i think it would be nice to suggest the use
of it instead of relying on IP layer fragmentation to make TCP flows more
middlebox friendly. In this draft. AFAIK, its the only option that does not
require new spec work, thats why it could make the cut for this draft.

For longer term architectural evolution of middlebox friendly IP fragmentation,
we could indeed have a new IP option carrying that context, and fragmentation
would put this option into every fragment. The definition of this context could
be per-transport proto.

I think there could be better middlebox contexts better than port numbers,
but to make fragments work better for existing TCP/UDP middlebox
functions, those 32 bits are it. But given how we can expect exposure of
information only from willing higher layers, they will have a much easier
way to get what they want to support by packet layer fragmentation. A
simple generic packet layer fragmentation for UDP would therefore be nice IMHO
so that UDP applications wanting to be friendly would not have to
reinvent that wheel.

If we actually ever do such an IP option, it MUST be a destination option,
because the insufficient RFCs defining the treatment of hop-by-hop options
burned any ability to deploy those.

For IPsec, IP in IP or similar higher layer protocols, i would either
use them as the key beneficiary of generic UDP fragmentation (IPsec/IP-UDP-IP)
for pragmatic short term solutions, or else the IP option would equally
be applicable to them (interesting discussionw aht the best context for
them would be, but the two port numbers would make them be most compatible
with those typcialyl very TCP/UDP centric middlebox functions).

Specific answers to your points below

Cheers
    Toerless

On Sun, Aug 26, 2018 at 08:01:07PM -0700, Joe Touch wrote:
> IPv6. IP options.

IMHO hop-by-hop optsion got burned and are undeployable because the
RFCs never made it mandatory enough to have them never impact forwarding
performance randonly and badly. We had to abandon perfectly good
hop-by-hop inspection solutions because there whee so many stupid router
implemntations out there that didn't even have the feature but would still
punt those packets to slow-path, then the operators saw those packets as
DoS packets and filtered them. 

> And (perhaps) any new proposed solution.

The main issue of everything written on top is that the main
business interests the IETF works against is that of non-middlebox
friendly participants.

> We have to aim at what network components *need* to do to participate. IP fragmentation is exactly that.

See above. The technical question is how to enable sufficient per-packet context.

> That???s easy to say, but since any host might be an IPsec tunnel endpoint, you???re back where we are now - needing IP fragmentation support everywhere, ultimately.

asked and answered.

> My view is simple - if you fix what we KNOW is wrong with NATs and firewalls - in KNOWN ways - then not only don???t we have to solve this problem, we don???t have a lot of other problems either (e.g., lack of state when a flow takes a different path into an enterprise).

Thats an orthogonal discussion. The goal in this fragmentation
thread is purely to eliminate virtual-reassembly complexity on
middleboxes. However good or bad it is what they do.

Actually, its not fully orthogonal. By eliminating virtual
reassembly needs, we make the middlebox also work for
cases where fragments use different paths.

> You???re optimizing a long-term impact solution for a short-term limitation. That???s a bad idea; protocols last a very long time.

The diference between per-packet operation and virtual-fragment-reassembly
is orders of magnitude of complexity. Its the same order of magnitude
a business problem for network device development as those unnecessary RTTs
in pre-QUIC transport are for companies trying to make make money
from ADD millenials on web pages.

> I am fine with encouraging the *search* for new solutions, as long as *in the meantime* we also call out firewalls and NATs for how they are already broken. Until IP fragmentation is deprecated, that has to be our position as a community.

Probably easier trying to figure out what subset of NAT/FW/middleboxes
we'd find to be worthwhile enough to support explicitly. Especially
for firewalls, many business interests say NONE and they will only
revert position when e.g.: they fail to get enough market share
with that position.

> Again, please tell me how to do that for IPsec tunnels - which, again, can start/end anywhere in the network.

See above. I was talking about the pragmatic approach for consenting hosts and
middleboxes to get away without having to enhance IP.

> Agreed, but that???s exactly where you???re headed when you say ???kick the can down the road to the upper layer protocol???.

I am still not sure about the meaning of your answers to Ole, but i
may have tried to argue on this point ("i like SOCKS") maybe in
the direction of how you may view middleboxes in your paper, e.g.: doing a lot
more higher layer. But thats a much larger story independent of the
problem of virtual fragment reassembly, which i think should be the
only relevant issue in this fragmentation specific thread (and
not the general bitching or improvments on firewall semantics).

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Aug 2018 15:51:58 -0700

Toerless,

I think it's the opposite-- the definition of the context should be
protocol agnostic. We need to get middleboxes out of doing DPI and to
stop worrying only about select transport protocols. So we need a
mechanism  that works equally well with with TCP, UDP, SCTP, ICMP,
IPsec, fragments, etc. It definitely needs to be secure though.

> so that UDP applications wanting to be friendly would not have to

That's already in UDP options and some UDP encapsulations like GUE.
It's a good idea, but doesn't completely obsolete the use of IP
fragmentation.

> burned any ability to deploy those.
In PANRG when I suggested that FAST could be done in a Destination
Option there was a lot of push back. I think it was for good reason.
Hop-by-Hop were designed precisely for inspection and potential
modification at intermediate nodes, and the requirement that all nodes
in the path process HBH has been relaxed in RFC8200. Destination
Options (as well as Fragment EH) aren't supposed to even be inspected
at intermediate nodes. The rationale for using DestOpts is of course
that they're less likely to be dropped by intermediate nodes. That's
true, they are more likely to be dropped; per RFC7872 it's about
15-17% drop rate for DestOpts and  40-43% for HBH. However, given the
update in RFC8200 and if some useful HBH options are defined, I would
expect that new deployments and replacements might start to lower the
HBH drop rate. In any case, the drop rates for DestOpts are still no
where close to zero, so regardless of which option is used used some
backoff is needed when the options are dropped to continue to work but
in a potentially degraded service mode relative to what a working
option could provide.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 29 Aug 2018 02:24:30 +0200

> IPsec, fragments, etc. It definitely needs to be secure though.

Sure, i meant to imply that port-numbers are useful pragmatically,
but other context identifiers would long term be better. 
Demux-Identifiers at the granualarity of a subscriber or 
application wold be a lot more scalable than flow identifiers.

Security is a wide topic. The firewall function of permitting return
traffic on a flow for internally initiated flows for example is a
wonderful simple function that in most deployment does a fine job
without additional security. And in a lot of embedded/walled-garden
networks, additionals ecurity throgh e.g.: ACLs (like through MUD)
is a more appropriate solution than cryptographic security. So
a bit more exploration of viable options would be useful. The least
i want to do is to force Internet PKI and complex out-of-band
middlebox discovery on all solutions where it's not needed.

> That's already in UDP options and some UDP encapsulations like GUE.

Pointers ?

> It's a good idea, but doesn't completely obsolete the use of IP

Nothing pragmatic will. Just a possible part of recommendations.

> Option there was a lot of push back. I think it was for good reason.

WHat was for good reason ? Your proposal or the pushback ?

> in the path process HBH has been relaxed in RFC8200.

Hop-by-hop options have been burned as i said through bad
implementations that for example punt them. Thats why operators often
configure to drop packets with those options to avoid them
burning their bad routers. 

Lets say we come up with some good new solution that depends on
new code written. I would hope we can document/standardize this
in such a way that new code would not be subject to this
legacy problem. But we can not get the benefit of that new code when
we use the existing burned code point for hop-by-hop because
we can not expect to get EXISTING code fixed and we will always
have paths with such old code.

Aka: if the religious architecture faction makes a fuzz out of
not using destination options for onpath functions, then lets
consider that we could simply rev the codepoint for hop-by-hop
options, but to make that solution stick, we would have to
show that we can write up correct processing RFCs for that
gen2 code-point such that it will not again get burned like
the first odepoint through bad new code.

I FOR ONCE WOULD SUGGEST TO WRITE THAT RFC STANDARDIZING THAT
REV2 HOP-BY-HOP CODEPOINT LIKE A VERY OLD RFC IN ONLY
CAPITAL LETTER TO GET IT INTO THE HEAD OF EVERY LAST 
IMPLEMENTOR OF THAT GEN2 HOP-BY-HOP OPTION CODE POINT TO
NOT REPEAT THE STUPIC CODE THEY WROTE IN ROUND 1.

Sorry for the soapbox, it's just been a frustration of
mine since  the early 2000 when the IPv6 specs did not
well enough improve on this point vs. IPv4 and i had
to rant about a lack of focus on reality of implementation
deltails to the IPv6 architects.

*sigh*.

> in a potentially degraded service mode relative to what a working

Yes, its a darn unthankful job trying to come up with good
architectures for middleboxes when three is so much bad code and
bad operations out there.

Hence the thought in another venue
to have those middlebox instructions encrypted in such a
way that any onpath node not explicitly trusted to process
would have no way to even determine whether those options are there
and hence had no way to figure out what packets to drop. Of course
this approach is a lot more expensive for forwarding planes and
does require a lot more out-of-band synchronization.

See above: short of something that extreme, we should focus
on doing the right thing for new code but build it in
such a way that it is not blocked by bad existing old onpath code.

Toerless

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Tue, 28 Aug 2018 18:02:19 -0700

>> That's already in UDP options and some UDP encapsulations like GUE.
https://tools.ietf.org/html/draft-ietf-tsvwg-udp-options
https://tools.ietf.org/html/draft-ietf-intarea-gue-extensions

> WHat was for good reason ? Your proposal or the pushback ?
The pushback was reasonable. Using Destination Options in this way
would be a hack and non-conformant with the standard.

> deltails to the IPv6 architects.
Per RFC8200:

"NOTE: While [RFC2460] required that all nodes must examine and
process the Hop-by-Hop Options header, it is now expected that nodes
along a packet's delivery path only examine and process the Hop-by-Hop
Options header if explicitly configured to do so."

That allowance should be sufficient to resolve most of the the
Hop-by-Hop being dropped problem. All that is being asked of
middleboxes is that they ignore HBH instead of dropping the packet if
they don't want to deal with them. That should not be difficult to
implement. Hopefully, it's just a matter of evangelization and time to
improve the situation.

> does require a lot more out-of-band synchronization.

Yes. But if this is done right, it would eliminate the need to
maintain costly flow state in the network. See
https://tools.ietf.org/html/draft-herbert-fast.

> such a way that it is not blocked by bad existing old onpath code.
Agreed.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 29 Aug 2018 04:10:15 +0200

> https://tools.ietf.org/html/draft-ietf-intarea-gue-extensions

Thanks

> Options header if explicitly configured to do so."

The industry does not fix these type of issues in old running code just
because someone else wants to deploy new code in new boxes somewhere
else along the path. That broken code will kill deployments for
a decade until it expires.  Besides: Anybody who continues to just
evolve code from an old code basis is also unlikely to read, understand
and fix this piece in the evolution to rfc8200 too.

And there is not even a MUST in that text. And it does not talk about
granularity of inspection. Guess what triggered the first wave of bad
implementations killing onpath services: IPv4 router alert that
was punting ANY IPv4 router alert. Instead of at least only per-IPv4
protocol field in the router-alert.

Sorry. Will never fly in real deployment across Internet paths
if its based on this lame text in rfc8200. I stand by my suggestion.

There is also no consideration at all to rethink even expressing
the granularity of deciding whether or not to inspect. 

> implement. Hopefully, it's just a matter of evangelization and time to

Good luck. Where can i buy short sell options for anything
done on the existing hop-by-hop code point ?

Cheers
    Toerless

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Aug 2018 08:07:23 -0700

Hi, Toerless, 

Overall, I think that it's OK for the doc to remind of us of what is
*already* required and best practice: 

- IPv4 hosts SHOULD avoid enabling in-net fragmentation (needed, in
part, for IP ID compliance at high rate per RFC 6864) 

- IP routers MUST support forwarding of fragments 

    - a router that forwards on information not available in the first
fragment is acting as a proxy for a host; in that case, it SHOULD
maintain enough context to forward fragments (which may involve delaying
non-initial fragments and/or keeping state equivalent to reassembly
between fragments) 

- NATs MUST support forwarding of fragments in both directions because
they act as proxies for the private side host (done in the same manner
as for routers, noted above) 

- transport layers SHOULD use PLPMTUD rather than PMTUD, if
fragmentation is supported at the transport layer  

IMO, all other recommendations are premature at best.  

Pushing transport IDs into the IP header doesn't help; the transport
layer already has EXACTLY the extra state needed (the frag ID), and new
IP options are both costly and not robust -- adding them at best
"shuffles the deck chairs". 

Joe

> You???re optimizing a long-term impact solution for a short-term limitation. That???s a bad idea; protocols last a very long time.

The diference between per-packet operation and
virtual-fragment-reassembly
is orders of magnitude of complexity. Its the same order of magnitude
a business problem for network device development as those unnecessary
RTTs
in pre-QUIC transport are for companies trying to make make money
from ADD millenials on web pages.

> Again, please tell me how to do that for IPsec tunnels - which, again, can start/end anywhere in the network.

See above. I was talking about the pragmatic approach for consenting
hosts and
middleboxes to get away without having to enhance IP.

> it becomes certainly difficult to even model this well. When you find a complete model better than the Internet, propose it.
> Until then...

HTTPs over DWDM with application layer proxies on every hop.
You didn't define how to measure "better" ;-) 
Agreed, but that???s exactly where you???re headed when you say ???kick
the can down the road to the upper layer protocol???. 
I am still not sure about the meaning of your answers to Ole, but i
may have tried to argue on this point ("i like SOCKS") maybe in
the direction of how you may view middleboxes in your paper, e.g.: doing
a lot
more higher layer. But thats a much larger story independent of the
problem of virtual fragment reassembly, which i think should be the
only relevant issue in this fragmentation specific thread (and
not the general bitching or improvments on firewall semantics).

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Aug 2018 08:11:44 -0700

On 2018-08-28 17:24, Toerless Eckert wrote:

> ...Sure, i meant to imply that port-numbers are useful pragmatically,

There are many problems with this issue. 

First, the reason that port numbers would be needed is that they are
*currently* how NATs demux, firewalls enforce policy, and routers manage
flows. For each of these, a different identifier could be developed, but
they would not then reduce the need for ALL of these at the IP level at
some boxes. E.g., see draft-touch-tcpm-sno 

Ultimately, we have to admit that a device that acts on behalf of a host
IS a host and costs what a host costs. 

We can't keep believing there is magic dust that can establish a
solution otherwise. 

Joe

---

From: tom <[email-masked]>
Date: Wed, 29 Aug 2018 09:53:50 -0700

> *currently* how NATs demux, firewalls enforce policy, and routers manage

There is no requirement in IP that all packets have a transport layer
header that with port numbers. In fact, there are several that don't
have them like IPsec, fragmented packets, or ICMP. Firewalls that drop
anything but UDP and TCP are doing nothing more that ossifying the
Internet. Besides that, there are now better alternatives anyway.
Reassembly for NAT doesn't require any knowledge or port numbers or
even what the transport protocol is. Routers can now use the IPv6 flow
label instead of expensive DPI to get transport ports for ECMP as all
major OSes now set non-zero flow labels.

> they would not then reduce the need for ALL of these at the IP level at some
> boxes. E.g., see draft-touch-tcpm-sno
>
> Ultimately, we have to admit that a device that acts on behalf of a host IS
> a host and costs what a host costs.

That in turn breaks the the end-to-end model. Middleboxes that attempt
to participate transport protocols, like a host, inevitably break
things and hence is another source of ossification. This is readily
evident apparent in that they can't participate in end-to-end crypto.
Of course they have tried to insert themselves into that realm, but
then we get abominations such as the forced MITM attacks of SSL
inspection. IMO, real end-to-end security is a core requirement that
outweighs any tradeoffs we might make for the security benefits of
firewalls.

>
> We can't keep believing there is magic dust that can establish a solution
> otherwise.
>
As they say, the answer to ossification is encryption. It does a long
way to at least force the issue. First TLS negated payload inspection
at middelboxes. QUIC's encryption of transport layer header
subsequently negated intermediate devices ability to peruse the
transport layer. But, crypto isn't magic dust either. There is only so
much of the packet we can encrypt, and a host at its discretion MAY
want to share to share transport layer information in the network to
get better service. So we need to keep working on solutions.

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Aug 2018 10:10:24 -0700

Tom,

On 2018-08-29 09:53, tom wrote:

> header that with port numbers. ...

Yes, we agree. It's not the only way they SHOULD or COULD work, but it
is how they DO work. 

> That in turn breaks the the end-to-end model.

Acting like what you are doesn't break anything; it lets you act to the
fullest extent possible. 

Relaying info through hosts inside a network path is what breaks the E2E
model - agreed. 

All I am saying is that: 
- IF you deploy a middle box, THEN it MUST act as a host and reassemble
(or do the equivalent) 

I wasn't endorsing the IF. 

> Middleboxes that attempt

They can* participate in crypto, but then the definition of E2E ends
where it should - at the middlebox. 

* = only if they somehow are given the key, of course 

> outweighs any tradeoffs we might make for the security benefits of

I would argue that it is OK to give a middlebox the key if that's OK for
a given trust model, e.g., it would make sense inside an enterprise to
offload security to the ingress of that enterprise. But not elsewhere; 

> As they say, the answer to ossification is encryption.

It's not an answer; it renders the question irrelevant, as it should. 

Not all questions necessarily have answers. As Rocket will tell you
(ref: end scenes, Guardians of the Galaxy), wanting something does not
make it so. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 29 Aug 2018 10:38:42 -0700

On Wed, Aug 29, 2018 at 10:10 AM, Joe Touch <[email-masked]> wrote:
> Yes, we agree. It's not the only way they SHOULD or COULD work, but it is
> how they DO work.
>
>
>
>
>
> - IF you deploy a middle box, THEN it MUST act as a host and reassemble (or
> do the equivalent)
>

I don't think you need the part about acting as a host, that would
have other implications. Also, the reassembly requirement might be
specific to NAT and not other middlebox functionality. For instance,
it would be sufficient for a firewall that is dropping UDP packets to
some port to only drop the first fragment that has UDP port numbers
and let the other fragments pass. Without the first fragment
reassembly at the destination will simply timeout and the whole packet
is dropped.

> They can* participate in crypto, but then the definition of E2E ends where
> it should - at the middlebox.
>
> I would argue that it is OK to give a middlebox the key if that's OK for a
> given trust model, e.g., it would make sense inside an enterprise to offload
> security to the ingress of that enterprise. But not elsewhere;
>
Sure enterprises can do that. But I'm more worried about the five
billion mobile devices that may connect to random WIFI or mobile
networks over the course of a day. For them there is simply no concept
that the network will provide any level of security.

Tom

> Not all questions necessarily have answers. As Rocket will tell you (ref:
> end scenes, Guardians of the Galaxy), wanting something does not make it so.
>
> Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Aug 2018 17:32:43 -0700

On 2018-08-29 10:38, Tom Herbert wrote:

> have other implications.

It does, and that's exactly why you do. In particular, this includes
ICMP processing. 

> Also, the reassembly requirement might be

And that's a great example of why not reassembling (or equivalent) isn't
the appropriate behavior. 

Yes, the packet will still not be delivered, but the receiver will end
up doing a lot of work that isn't necessary. I.e., the middlebox has
ignored work it was responsible for and caused work elsewhere.  

Further, acting as a host is always the right thing for any node that
sources packets with its own IP address -- that includes NATs and
regular proxies. The behavior of transparent proxies is more complex,
but can be similarly reasoned from the appropriate equivalence model. 

> that the network will provide any level of security.

Which is a great reason why it might actually be useful to shift the
security work to a "home entrance" device by placing the security there.
It's a matter of system configuration, but the point is that it isn't
the device that makes it incorrect; it is how it is configured and
deployed. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 29 Aug 2018 18:34:18 -0700

On Wed, Aug 29, 2018 at 5:32 PM, Joe Touch <[email-masked]> wrote:
> It does, and that's exactly why you do. In particular, this includes ICMP
> processing.
>
>
> And that's a great example of why not reassembling (or equivalent) isn't the
> appropriate behavior.
>
> Yes, the packet will still not be delivered, but the receiver will end up
> doing a lot of work that isn't necessary. I.e., the middlebox has ignored
> work it was responsible for and caused work elsewhere.

Joe,

End hosts are already quite capable of dealing with reassembly, I
think you'll find the average middlebox is not prepared to handle it.
In truth, for this case it really doesn't save the hosts much at all.
A DOS attack on fragmentation is still possible by the attacker
sending all but the last fragment to a port that is allowed by the
firewall. Also, a destination host will receive all the fragments for
reassembly by virtue of it being the having destination address in the
packets. As discussed previously, there's no guarantee that a firewall
will see all the packets in a fragment train in a mulithomed
environment-- routing may take packets along different paths so they
hit hit different firewalls for a site. The answer to that seems to be
to somehow coordinate across all the firewalls for a site to act as
single host-- I suppose that's possible, but it would be nice to see
the interoperable protocol that makes that generally feasible at any
scale.

> sources packets with its own IP address -- that includes NATs and regular
> proxies. The behavior of transparent proxies is more complex, but can be
> similarly reasoned from the appropriate equivalence model.

Proxies aren't quite the same though. An explicit proxy at least is
both receiving and sourcing packet based on it's own address. NAT only
sources or receive packets with their own address half the time.
Firewalls, never do and don't even need a host address.

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Aug 2018 19:58:06 -0700

On 2018-08-29 18:34, Tom Herbert wrote:

> Joe,
> 
> End hosts are already quite capable of dealing with reassembly,

Regardless, middleboxes shouldn't be avoiding their own effort by
creating work for others. A corollary to the Postal Principle should be
"you make the mess, you clean it up".  

FWIW, the idea of dumping just the first fragment and letting the
receiver clean it up was tried in ATM in the late 1980s and it failed
badly. It turns out that congestion isn't always a point problem - when
multiple routers in a path are overloaded (which can and does happen),
not dropping the rest of the fragments can cause downstream congestion
that wouldn't have happened otherwise and then drops to other "real"
packets. 

> I

Sure, but that's a problem I'm hoping we can fix rather than encourage
continued bad behavior. 

> In truth, for this case it really doesn't save the hosts much at all.

It won't prevent endpoint attacks, but it does mitigate the effect of
useless fragment processing. And, as per above, it avoids drops to other
packets that could/should have made it through. 

> the interoperable protocol that makes that generally feasible at any

Compared to other solutions proposed in this thread, that one is nearly
trivial to design. The issue is having operators - who deploy these
devices in ways that they should know need this feature - enable it
properly (i.e., point them all at each other). 

> Proxies aren't quite the same though.

They are three different things, as noted in the paper I posted earlier,
but they all are variants of requiring host behavior of some sort. 

> An explicit proxy at least is

Sure, but there's more to it than just using the address...(see next
note) 

> Firewalls, never do and don't even need a host address.

Transport protocols are endpoint demultiplexers and state managers;
anything that uses that info and/or state is also acting as a host and
needs to follow at least some host requirements too (all that apply to
transports, including translation of signaling related to transport
protocols and ports). 

Joe

---

From: Christian Huitema <[email-masked]>
Date: Wed, 29 Aug 2018 23:19:30 -0700

On 8/29/2018 7:58 PM, Joe Touch wrote:
> creating work for others. A corollary to the Postal Principle should
> be "you make the mess, you clean it up". 

Joe's stubborn adherence to the letter of the RFC would be very nice if
the protocol police could somehow punish the merchants of NATs... The
whole discussion reminds me of Martin Thomson's draft, "use it or lose
it" (draft-thomson-use-it-or-lose-it-02). Martin is describing how
extension mechanisms that are not actually used get ossified away by the
deployment of middle-boxes. The same happened long ago with IP
segmentation. With NATs, applications cannot assume that reassembly will
work. With Firewalls, transports cannot assume that ICMP will work.

-- Christian Huitema

---

From: Tom Herbert <[email-masked]>
Date: Thu, 30 Aug 2018 07:56:24 -0700

On Wed, Aug 29, 2018 at 7:58 PM, Joe Touch <[email-masked]> wrote:
> Regardless, middleboxes shouldn't be avoiding their own effort by creating
> work for others. A corollary to the Postal Principle should be "you make the
> mess, you clean it up".
>
> FWIW, the idea of dumping just the first fragment and letting the receiver
> clean it up was tried in ATM in the late 1980s and it failed badly. It turns
> out that congestion isn't always a point problem - when multiple routers in
> a path are overloaded (which can and does happen), not dropping the rest of
> the fragments can cause downstream congestion that wouldn't have happened
> otherwise and then drops to other "real" packets.
>
>
> I
> trivial to design. The issue is having operators - who deploy these devices
> in ways that they should know need this feature - enable it properly (i.e.,
> point them all at each other).
>
Joe,

I would be amazed if firewall vendors consider this "nearly trivial to
design". Reassembly requires memory to hold packets, a non-work
conserving datapath, requires state to be maintained, and the
aforementioned problems of consistent routing of fragments needs to be
resolved. A middlebox would be performing reassembly on behalf of some
number of backend hosts, so the memory requirement for reassembly is
some multiplier of that needed by an individual host. Non-work
conserving means packets need to be queued at the device which
requires cache management and introduces delay. Requiring state in
_stateless_ devices is a problem, it's likely they have neither the
mechanisms nor the memory to support reassembly. And then there's the
Denial Of Service considerations... the middlebox is now an obvious
target for DOS attack on reassembly. We need to deal with this on
hosts, but the attacks are going to be worse on middleboxes. Consider
that a middlebox wouldn't normally know all possible hosts in the
network, so it may very well end up reassembling packets for
destinations that don't even exist! And on top of all of this,
applications are still motivated to avoid fragmentation for other
reasons, so I suspect vendors will view as a lot of work for very
little benefit.

> They are three different things, as noted in the paper I posted earlier, but
> they all are variants of requiring host behavior of some sort.
>
>
>
> Sure, but there's more to it than just using the address...(see next note)
>
>
>
> Transport protocols are endpoint demultiplexers and state managers; anything
> that uses that info and/or state is also acting as a host and needs to
> follow at least some host requirements too (all that apply to transports,
> including translation of signaling related to transport protocols and
> ports).

Maybe so, but at best middleboxes can only approximate host behavior.
Requiring them to perform reassembly is only addressing one symptom of
the disease. The real disease is intermediate devices that try to
insert themselves into transport layer protocols by DPI or trying to
infer transport layer state. Calling them "hosts" doesn't change the
fact that such devices will break the end-to-end model and ossify the
Internet. As they say, "You can put lipstick on a pig, but it's still
a pig!" :-).

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 30 Aug 2018 10:31:03 -0600

> On Aug 30, 2018, at 8:56 AM, Tom Herbert <[email-masked]> wrote:
> 
> design".

The coordination protocol is, and that’s all I claimed. 

> Reassembly requires memory to hold packets, a non-work
> _stateless_ devices is a problem,

It would if they were, but they’re not. So I’m suggesting these devices need to add more state and work to clean up the mess they make. 

> it's likely they have neither the

As they did for devices that don’t support v6. 

> Internet. As they say, "You can put lipstick on a pig, but it's still

That’s their problem and will increasingly render them less useful over time. 

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 30 Aug 2018 17:26:32 -0700

> On Aug 29, 2018, at 11:19 PM, Christian Huitema <[email-masked]> wrote:
> 
>> Regardless, middleboxes shouldn't be avoiding their own effort by creating work for others. A corollary to the Postal Principle should be "you make the mess, you clean it up". 
> 
> Joe's stubborn adherence to the letter of the RFC would be very nice if the protocol police could somehow punish the merchants of NATs…

My concerns are pragmatic - merely not supporting something does not make it unnecessary.

There was a time when Internet service in hotels would regularly block all except basically DNS and HTTP/HTTPS; that’s becoming much less the case. There was a time when devices didn’t support IPv6 at all because it was considered merely an unnecessary expense; that’s becoming much less the case too.

In this case, we have two problems 
	1) NATs/firewalls as currently implemented do not support fragments
	2) our current protocols, in many cases, require fragments (IPsec tunnel mode) and in other cases (tunnels in general) would benefit from IP fragmentation support
	3) we DO NOT HAVE an alternative (there are some piece-wise proposals for various aspects, but none support IPsec tunnel mode and none are otherwise universal

Yes, something needs to be done, but I argue that *until we have a worked alternative*, we need to keep restating the fact - NATs/firewalls MUST reassemble to work properly; where they don’t, the error is on them - not the rest of the Internet for using fragments.

> The whole discussion reminds me of Martin Thomson's draft, "use it or lose it" (draft-thomson-use-it-or-lose-it-02). Martin is describing how extension mechanisms that are not actually used get ossified away by the deployment of middle-boxes. The same happened long ago with IP segmentation. With NATs, applications cannot assume that reassembly will work. With Firewalls, transports cannot assume that ICMP will work. 

Yes, that’s the tension:
	a) identify bugs and fix them
	b) accept bugs as de-facto protocol evolution

The problem with (b) is that it is not guided by what Internet users need; it’s guided by what is profitable for individual vendors. That path is hazardous - there’s no reason to believe that the result will be a useful Internet. So until we know it’s safe to do (b), we need to stick with (a).

Joe

---

From: Christian Huitema <[email-masked]>
Date: Thu, 30 Aug 2018 19:56:15 -0700

On 8/30/2018 5:26 PM, Joe Touch wrote:
>
>
>> On Aug 29, 2018, at 11:19 PM, Christian Huitema <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
>> Joe's stubborn adherence to the letter of the RFC would be very nice
>> if the protocol police could somehow punish the merchants of NATs…
>
> There was a time when Internet service in hotels would regularly block
> all except basically DNS and HTTP/HTTPS; that’s becoming much less the
> case. There was a time when devices didn’t support IPv6 at all because
> it was considered merely an unnecessary expense; that’s becoming much
> less the case too.
>
> 2) our current protocols, in many cases, require fragments (IPsec
> tunnel mode) and in other cases (tunnels in general) would benefit
> from IP fragmentation support
> 3) we DO NOT HAVE an alternative (there are some piece-wise proposals
> for various aspects, but none support IPsec tunnel mode and none are
> otherwise universal
>
> Yes, something needs to be done, but I argue that *until we have a
> worked alternative*, we need to keep restating the fact -
> NATs/firewalls MUST reassemble to work properly; where they don’t, the
> error is on them - not the rest of the Internet for using fragments.
>
>> The whole discussion reminds me of Martin Thomson's draft, "use it or
>> lose it" (draft-thomson-use-it-or-lose-it-02). Martin is describing
>> how extension mechanisms that are not actually used get ossified away
>> by the deployment of middle-boxes. The same happened long ago with IP
>> segmentation. With NATs, applications cannot assume that reassembly
>> will work. With Firewalls, transports cannot assume that ICMP will work. 
>
> Yes, that’s the tension:
> The problem with (b) is that it is not guided by what Internet users
> need; it’s guided by what is profitable for individual vendors. That
> path is hazardous - there’s no reason to believe that the result will
> be a useful Internet. So until we know it’s safe to do (b), we need to
> stick with (a).

Shouting from the hilltops is not sufficient. We can very well do it,
but our voice is not very loud, and our hill is not very tall. Fixes
happen when there is some customer pressure, as in "your bug blocks some
application that many customers love." I just don't see any application
putting pressure on NATs and firewalls to support fragmentation. HTTP,
email, or Web RTC seem to be working just fine. It may be that IPSEC and
some tunnels do not work in theory, but VPNs do work in practice. It may
be that protocol stacks could clean up some of their kludges if
fragmentation worked, but the market does not care a whole lot about
that. And without such pressure, the protocol police is not going to be
very effective.

By the way, the biggest issue is not so much fragmentation as
black-holing. One thing is for a NAT to not support reassembly. Another
is to get a fragment and just drop it without telling anything to
anyone. Or even worse, forward an initial-but-incomplete fragment and
drop the other fragments, all that without telling anything to anyone.
Strategically, that may well be where to put the pressure. Reassembly
requires state and memory, but sending an ICMP at most requires some
form of rate control. That should not be too hard.

Of course, then we will have to convince firewalls to not drop these
ICMP packets. But there the dynamics are manageable, because the
firewalls dropping the ICMP and the servers that could benefit from them
are often managed by the same folks. So maybe there is some hope after all.

-- Christian Huitema

---

From: Tom Herbert <[email-masked]>
Date: Fri, 31 Aug 2018 08:44:25 -0700

On Thu, Aug 30, 2018 at 5:26 PM, Joe Touch <[email-masked]> wrote:
> Joe's stubborn adherence to the letter of the RFC would be very nice if the
> protocol police could somehow punish the merchants of NATs…
>
>
> There was a time when Internet service in hotels would regularly block all
> except basically DNS and HTTP/HTTPS; that’s becoming much less the case.
> There was a time when devices didn’t support IPv6 at all because it was
> considered merely an unnecessary expense; that’s becoming much less the case
> too.
>
> 2) our current protocols, in many cases, require fragments (IPsec tunnel
> mode) and in other cases (tunnels in general) would benefit from IP
> fragmentation support
> 3) we DO NOT HAVE an alternative (there are some piece-wise proposals for
> various aspects, but none support IPsec tunnel mode and none are otherwise
> universal
>
Joe,

There is an alternative: don't use NAT! In a draft that is
recommending against using a core IP protocol feature like
fragmentation, I think it is entirely appropriate to recommend against
using the very features that are breaking it in the first place. IMO,
this draft should recommend people use of IPv6 without NAT to resolve
the problems with fragmentation caused by NAT.

> Yes, something needs to be done, but I argue that *until we have a worked
> alternative*, we need to keep restating the fact - NATs/firewalls MUST
> reassemble to work properly; where they don’t, the error is on them - not
> the rest of the Internet for using fragments.

Reassembly could only be a MUST for NAT, not firewalls. NAT might be
required because of the identifier space issue, however we already
shown how a firewall can achieve proper functionality without
reassembly and to be stateless by forwarding fragments and potentially
dropping the first one that contains port information being filtered.
The fact that this might forward fragments that are never reassembled
is at best an optimization with unproven benefit.

There is another case where in-network reassembly could be required
which is load balancing to a virtual IP address. This is handled in
the Maglev load balancer
(https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44824.pdf)
without employing a synchronization protocol. It is rather complex
though and not a solution easily generalized for simple environments.
Like NAT though, in the long run I believe IPv6 offers a better
solution that would eliminate the need for VIPs.

Tom

>
> The whole discussion reminds me of Martin Thomson's draft, "use it or lose
> it" (draft-thomson-use-it-or-lose-it-02). Martin is describing how extension
> mechanisms that are not actually used get ossified away by the deployment of
> middle-boxes. The same happened long ago with IP segmentation. With NATs,
> applications cannot assume that reassembly will work. With Firewalls,
> transports cannot assume that ICMP will work.
>
>
> Yes, that’s the tension:
> The problem with (b) is that it is not guided by what Internet users need;
> it’s guided by what is profitable for individual vendors. That path is
> hazardous - there’s no reason to believe that the result will be a useful
> Internet. So until we know it’s safe to do (b), we need to stick with (a).
>
> Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Aug 2018 08:56:55 -0700

> On Aug 31, 2018, at 8:44 AM, Tom Herbert <[email-masked]> wrote:
>> 
> Joe,
> 
> There is an alternative: don't use NAT!

Agreed - that should also be part of the observations of this doc.

> Reassembly could only be a MUST for NAT, not firewalls.

“or its equivalent"

> NAT might be

First, firewalls that port-filter need to do the same thing as a NAT in terms of keeping state.

> is at best an optimization with unproven benefit.

ATM proved otherwise in numerous published studies in the late 1980s. Those fragments compete for bandwidth further along the path; anytime they “win”, that decision is not work-conserving.

Note that keeping some state is already needed (if port-filtering) and that - as you note - the state filtering need not be “perfect”. However, it really ought to be SHOULD at least.

> which is load balancing to a virtual IP address. 

Any middlebox that uses state not available in all fragments MUST reassemble or keep equivalent storage/state to process fragments appropriately.

> solution that would eliminate the need for VIPs.

That’s true right up until we end up in a world where (mostly) nobody correctly uses flow IDs. Oh, wait - we’re already there…

Joe

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 31 Aug 2018 16:12:38 +0000

>> Any middlebox that uses state not available in all fragments MUST reassemble or keep equivalent storage/state to process fragments appropriately.

This statement is true without question, so the only question is what applications
would produce IP fragments that need to be forwarded by a middlebox. We have
already seen that ‘iperf3’ produces IP fragments by default on some systems. And,
intarea-tunnels makes the case for tunnels.

I will also make the case for NAT66. With RFC4193 ULAs, NAT66 will be inevitable,
but a middlebox may be able to translate based only on the IPv6 addresses and
not transport-layer port numbers. Such a middlebox could forward fragments
without having to reassemble or otherwise hold them until all fragments have
arrived.

Thanks - Fred

From: Int-area [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Friday, August 31, 2018 8:57 AM
To: Tom Herbert <[email-masked]>
Cc: int-area <[email-masked]>; Toerless Eckert <[email-masked]>; [email-masked]
Subject: Re: [Int-area] WG Adoption Call: IP Fragmentation Considered Fragile

On Aug 31, 2018, at 8:44 AM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:

Joe,

“or its equivalent"

NAT might be

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 31 Aug 2018 09:38:05 -0700

On Fri, Aug 31, 2018 at 8:56 AM, Joe Touch <[email-masked]> wrote:
> ATM proved otherwise in numerous published studies in the late 1980s. Those
> fragments compete for bandwidth further along the path; anytime they “win”,
> that decision is not work-conserving.
>
ATM from the 1980s? Is there any evidence that this is a real problem
impacting users in this century?

> Note that keeping some state is already needed (if port-filtering) and that
> - as you note - the state filtering need not be “perfect”. However, it
> really ought to be SHOULD at least.
>
>
> Any middlebox that uses state not available in all fragments MUST reassemble
> or keep equivalent storage/state to process fragments appropriately.
>
That requirement would include pretty much be applicable to every
router on the planet that does ECMP based on hashing transport ports.
Good luck fixing all of those to do reassembly :-)

> That’s true right up until we end up in a world where (mostly) nobody
> correctly uses flow IDs. Oh, wait - we’re already there…
>
Huh? Who is not correctly flow labels? Besides, in IPv6 there's plenty
of address space to that address sharing should be not needed and
routing is sufficient without looking beyond IP header.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 31 Aug 2018 09:49:06 -0700

> On Aug 31, 2018, at 9:38 AM, Tom Herbert <[email-masked]> wrote:
> 

Not until we instrument and measure these boxes. But that doesn’t mean a 30yr old KNOWN problem won’t repeat. 

> Good luck fixing all of those to do reassembly :-)

They need to either reassemble or keep state that equivalently let’s fragments share paths - or the don’t do their job as advertised. Fortunately if they’re just picking among paths that all work it won’t blackhole the traffic. 

> Huh? Who is not correctly flow labels?

Lots of endpoints don’t bother using them. 

> Besides, in IPv6 there's plenty

Address sharing is one reason NATs are used. Others are stateful firewalls (which also exist without NAT) and stateful forwarding (also exists without NAT). All require some level of work beyond the ‘cheap and dirty’ to actually get things correct. 

Joe