Subject: [Int-area] WGLC on draft-ietf-intarea-frag-fragile-05

From: Wassim Haddad <[email-masked]>
Date: Mon, 14 Jan 2019 18:13:42 +0000

Dear all,

This email starts an Int-Area WG Last Call on the latest version of "IP Fragmentation Considered Fragile” draft:
 
https://tools.ietf.org/html/draft-ietf-intarea-frag-fragile-05
 
Please respond to this email to support the document and/or send comments by 2019-01-28.
 
Please indicate if you are personally aware of any IPR that applies to draft-ietf-intarea-frag-fragile-xx? 
If so, has this IPR been disclosed in compliance with IETF IPR rules?

Regards,

Juan & Wassim

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 14 Jan 2019 14:55:23 -0500

I have re-read this document.  I think it is a useful document that 
captures that state of a complex tradeoff and makes effective 
recommendations. I support publishing it as a BCP.

If the authors make further additions, adding a mention of ECMP as a 
particular case of stateless load balancers might further improve the 
document.

Yours,
Joel

On 1/14/19 1:13 PM, Wassim Haddad wrote:
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 14:04:02 -0800

Hello. I have a couple of comments:

>From the draft:
"Middle boxes SHOULD process IP fragments in a manner that is
compliant with RFC 791 and RFC 8200.  In many cases, middle boxes must
maintain state in order to achieve this goal."

This requirement is confusing to me on several accounts. First of all,
there are a lot of requirements about fragmentation in both RFC791 and
RFC8200, including some MUSTs. This requirement seems to be updating
and possibly relaxing some of those requirements, but is not specific.
This seems ambiguous as a normative requirement.

Secondly, the only specified interaction between fragmentation and
intermediate nodes is that routers can fragment packets in IPv4. Other
than that, a middlebox that complies with RFC791 and RFC8200 does not
process or consider fragmentation of packets. Given that, it's unclear
to me why middle boxes would need to maintain state to be protocol
compliant. It's possible that the implicit exception of the
requirement is that middleboxes might perform "in-network reassembly"
or "virtual reassemlby" which would require state. If that is indeed
the case then the requirements for the mechanisms should be spelled
out.

For stateless load balancing (described in section 4.4), the IPv6 flow
label obviates the need for DPI. It is sufficient to hash over the
three tuple <saddr, daddr, flow label> to get good load balancing. All
major OSes have been updated to set flow labels, and there are devices
that already support this. IMO, the draft should make using flow label
for stateless load balancing a SHOULD.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 15 Jan 2019 14:29:53 +1300

On 2019-01-15 11:04, Tom Herbert wrote:
> This requirement is confusing to me on several accounts. 

Me too. I think the root of the problem is the word "compliant". To
be compliant with the IP model, middleboxes should not exist. I think
what the text is trying to say is more like:

"Middle boxes SHOULD process IP fragments in a manner that is
consistent with RFC 791 and RFC 8200."

That's a requirement for effective transparency, not for compliance.

> First of all,

Agreed, and the text would need to be a bit reorganized for that, by
separating the v4 and v6 cases completely. Also things are rather
different for in-transit load balancing (including ECMP and LAG) and
server load balancing, but I don't think that changes the recommendation.
The draft could cite RFC7098 for the server case. In RFC7098, we
assumed that the balancer could revert to the old DPI method if the flow
label is zero, which may lead to a fail if the packet is fragmented.
But for in-transit load balancing of large numbers of flows, the
{source, dest, flow_label} tuple will provide entropy even if the
flow label is zero.

  Brian

---

From: Tom Herbert <[email-masked]>
Date: Mon, 14 Jan 2019 17:44:06 -0800

> {source, dest, flow_label} tuple will provide entropy even if the

Right, if flow label is zero the algorithm just falls back to hashing
over the 2-tuple which is already what devices do today when then
can't parse the transport layer ports to get an L4 hash (e.g. for an
IPsec or GRE/IP packet).

Tom

---

From: Joe Touch <[email-masked]>
Date: Wed, 16 Jan 2019 22:20:37 -0800

Tom,

> and possibly relaxing some of those requirements, but is not specific.
I disagree; being compliant with existing RFCs neither updates nor
relaxes those RFCs.
> This seems ambiguous as a normative requirement.
Perhaps vague, agreed.
> intermediate nodes is that routers can fragment packets in IPv4.

Agreed. However, nodes that *create* packets with their own IP addresses
act as hosts; in that regard, middleboxes act both as intermediate (from
the private side to public) and hosts (from the public side). As hosts,
packets entering them, destined to *their* IP addresses, need to be
reassembled, just as with any other host.

This is addressed in detail here:

Touch, J,. "Middlebox Models Compatible with the Internet," Technical
Report, USC/ISI (ISI-TR-711), 2016.

https://www.strayalpha.com/pubs/isi-tr-711.pdf

> Other
> compliant. 
If they process only IP headers, they don't. If they further consume the
packet's contents - as would a *host*, they need to reassemble that
content (even if virtually) to be able to interpret it. That includes
associating transport headers with all the content as well as inspecting
that content.
> It's possible that the implicit exception of the
As with Ron, I agree that seems out of scope for this doc.
> label obviates the need for DPI. 
Only for flow-based routing; not for DPI processing involving content
beyond the transport header, e.g., content-based routing. I.e., it
depends on how far into the packet the DPI is going...
> It is sufficient to hash over the

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 17 Jan 2019 10:42:47 +0100 (CET)

> Please respond to this email to support the document and/or send comments by 2019-01-28.

I re-read the entire document as if I never read it before. Commenting as 
I am reading it:

2.1

"Therefore, the downstream router drops the packet and sends an
    Internet Control Message Protocol (ICMP) [RFC0792] [RFC4443] Packet
    Too Big (PTB) message to the source node.  The ICMP PTB message"

Yes, that's what it's supposed to do, and it does that unless it doesn't 
because something is wrong. Everything in 2.1 describes the way things are 
supposed to work in a language that just states "this is what happens". 
Wouldn't it make more sense to write it in a wording that is "this is 
supposed to happen" or "the design states that this should happen" or 
"this is expected behavior"?

The header about PMTUD characteristics is a bit short, it has lots of 
other characteristics. There are a lot of other things it relies on. Does 
this need to be enhanced somehow? Otherwise, change wording to make it 
explicit that these are just examples?

2.2

"   When an upper-layer protocol submits data to the underlying IP
    module, and the resulting IP packet's length is greater than the
    PMTU, the packet can be divided into fragments.  Each fragment
    includes an IP header and a portion of the original packet."

What about changing this to:

"   If an upper-layer protocol submits data to the underlying IP
    module, and the resulting IP packet's length is greater than the
    PMTU, the packet will be divided into fragments.  Each fragment
    includes an IP header and a portion of the original packet."

Since the whole idea of this document is to try to steer people away from 
using fragmentation, doesn't it make sense to word things in a way that 
states what will happen, and the "if" is more "please don't do this"?

"  [RFC0791] describes IPv4 fragmentation procedures.  An IPv4 packet
    whose DF-bit is set to one cannot be fragmented."

I thought the DF bit was instructions to the downstream routers and not to 
the host itself? What happens if I send an fragmented IPv4 packet with the 
DF bit set? Is this invalid?

RFC 791 says:

"An internet datagram can be marked "don't fragment."  Any internet
     datagram so marked is not to be internet fragmented under any
     circumstances."

Does this really prohibit the host from sending fragmented packets with 
the DF bit set?

4.3. Does this paragraph imply that a stateful firewall uses virtual 
re-assembly? Perhaps this should be stated explicitly? As an answer to 
"This problem does not occur in stateful firewalls." Perhaps 4.2 can be 
enhanced to talk about NAT/PAT and stateful firewalls (I imagine they 
share similar characteristics when it comes to handling fragmented 
packets?)

4.4. I'm sure there are other load balacing algorithms that uses all kinds 
of input, not only 5-tuple. Text should reflect that the use of 5-tuple is 
an example of an algorithm, not exhaustive list.

4.7

"Transient loss of ICMP PTB messages can cause transient black holes."

suggested new text:

"Transient loss of ICMP PTB messages can cause transient PMTU black 
holes."

Perhaps even the 4.7 header should be changed the same way to talk about 
"PMTU black hole" instead of just "black hole"

4.7.x. The case where a router doesn't generate PTB messages at high 
enough rate to handle all the flows. A few months back I helped diagnose a 
misconfiguration where MTU on an interface had been set to 150. That 
router wasn't able to generate PTB messages at the required rate to handle 
this situation.

4.8

I don't like the word "filtering". Filtering for me conveys intent. I have 
seen platforms that handle packets with fragment header differently than 
packets without fragment header, by default. So out of the box, this 
device had horrible forwarding performance for packets with fragmentation 
header. I'd like to have the word "filtering" here replaced with "loss" or 
similar word that doesn't convey intent. The main text already does this, 
but the 4.8 header says "filtering".

5.1

"While TCP will never cause the underlying IP module to emit a packet
    that is larger than the PMTU estimate, it can cause the underlying IP
    module to emit a packet that is larger than the actual PMTU.  If this
    occurs, the packet is dropped, the PMTU estimate is updated, the
    segment is divided into smaller segments and each smaller segment is
    submitted to the underlying IP module."

This is the ideal case and works well until it doesn't. If there is no 
PTB received and PLPMTUD isn't turned on, above will never happen and PMTU 
blackhole will persist.

7.4.

I'd like to start this paragraph with

"Operators MUST ensure proper PMTUD operation
    in their network"

... or similar text. This is the #1 thing for operators to do, and that 
implies for instance not filtering PTBs. So to put even more emphasis on 
this, I'd like to see this text re-structured to bring this requirement to 
the start of the paragraph. I believe I will be pointing to 7.4 for many 
years to come in discussions with fellow ISP colleagues.

Final comment:

I think this document is in good shape, valuable to the community, and I 
support it proceeding to publication with above comments.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 06:55:14 -0800

On Wed, Jan 16, 2019 at 10:20 PM Joe Touch <[email-masked]> wrote:
> I disagree; being compliant with existing RFCs neither updates nor relaxes those RFCs.
>
> Agreed. However, nodes that *create* packets with their own IP addresses act as hosts; in that regard, middleboxes act both as intermediate (from the private side to public) and hosts (from the public side). As hosts, packets entering them, destined to *their* IP addresses, need to be reassembled, just as with any other host.
>
> Touch, J,. "Middlebox Models Compatible with the Internet," Technical Report, USC/ISI (ISI-TR-711), 2016.
>
> If they process only IP headers, they don't. If they further consume the packet's contents - as would a *host*, they need to reassemble that content (even if virtually) to be able to interpret it. That includes associating transport headers with all the content as well as inspecting that content.
>
Joe,

As I mentioned, in-network reassembly has not been specified, only
reassembly at end destinations has been. I suspects that
implementations of in-network reassembly commonly make two incorrect
assumptions:

1) All fragments of a packet traverse the network device performing reassembly
2) The first fragment is always received first

Neither of these assumptions are valid for IP.

> Only for flow-based routing; not for DPI processing involving content beyond the transport header, e.g., content-based routing. I.e., it depends on how far into the packet the DPI is going...
>
TLS and payload encryption have largely eliminated the ability for
intermediate devices to perform DPI into content. Transport layer
header encryption, like done in QUIC, will further reduce the value of
doing DPI.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 07:06:14 -0800

Hi Tom,

> On Jan 17, 2019, at 6:55 AM, Tom Herbert <[email-masked]> wrote:
> 
> reassembly at end destinations has been.

Hint - if a packet arrives on your interface with your IP address, you ARE a host.

> I suspects that

The first one is above...
> Neither of these assumptions are valid for IP.

Agreed, but again, there are three incorrect assumptions. The first is that a NAT isn’t a host, and too often that’s the one that causes the above two to actually cause pain.

> header encryption, like done in QUIC, will further reduce the value of

Yes and no; there are still devices that do DPI on content - especially devices that spoof the initial splash page for Internet access when at hotspots. Some of them won’t even open a splash page unless customers first access a non-TLS website.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 07:27:22 -0800

Joe,

Conversley, if a packet arrives on your interface that isn't destined
to your IP address, you ARE NOT a host.

> Agreed, but again, there are three incorrect assumptions. The first is that a NAT isn’t a host, and too often that’s the one that causes the above two to actually cause pain.
If NAT is a host, then it is only a host in the return path. Packets
sent from the client behind a NAT have destinations addresses that are
not those of the the NAT device. If a device is performing host
procedures on such packets, like reassembly, then it is being done
outside the auspices of the standard.

In any case, this isn't a NAT specific issue. Stateful firewalls
regularly process packets for which they aren't the destination.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 08:24:48 -0800

Hi, Tom, 

On 2019-01-17 07:27, Tom Herbert wrote:

> ...
 Joe,

Correct. Note that the association of host with traffic counts for both
transmission and reception, though. 

> Agreed, but again, there are three incorrect assumptions. The first is that a NAT isn't a host, and too often that's the one that causes the above two to actually cause pain.

Not quite. The return path is certainly easier to see, but the forward
path doesn't *relay* the incoming packets; it consumes them. Which is
what a host does. 

The outgoing packets are emitted with the source IP of the NAT. So in
the forward direction, strictly, only the *content* is relayed. 

If that content is modified (i.e., address/port rewriting) or inspected,
then that device is acting as a host. 
If the outgoing HOST packets depend on something of the incoming ones,
then that device is also acting as a host. 

> regularly process packets for which they aren't the destination.

And they are similarly bound by the rules of being a host *when they
do*. 

The Internet requirements work just fine; its the devices that aren't
following them. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 08:58:53 -0800

On Thu, Jan 17, 2019 at 8:24 AM Joe Touch <[email-masked]> wrote:
> Correct. Note that the association of host with traffic counts for both transmission and reception, though.
>
>
>
> I suspects that
> Not quite. The return path is certainly easier to see, but the forward path doesn't *relay* the incoming packets; it consumes them. Which is what a host does.
>
> The outgoing packets are emitted with the source IP of the NAT. So in the forward direction, strictly, only the *content* is relayed.
>
> If that content is modified (i.e., address/port rewriting) or inspected, then that device is acting as a host.
> If the outgoing HOST packets depend on something of the incoming ones, then that device is also acting as a host.
>
>
>
> And they are similarly bound by the rules of being a host *when they do*.
>
Joe,

That's impossible. The first rule is that a host is defined as a node
to which packets are addressed. Intermediate nodes performing host
processing on packets that don't have their address have already
violated rule #1. If they're keeping flow state in the network
(connection state or reassembly), then that inevitably forces the
requirement that packets have to be routed through a particular node
in order for state to be maintained. This is not a problem hosts have,
and it's not required in IP that packets of a flow have to always take
the same path to a destination. Without additional requirements on
deployment, stateful devices break multi-path.

So if these devices need to *emulate* host behaviors then there really
should be a specification to that effect. It's not enough to simply
say they have to follow host requirements. They can't and they clearly
impose additional requirements.

Tom

> The Internet requirements work just fine; its the devices that aren't following them.
>
> Joe
>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 12:48:30 -0800

Hi, Tom,

On 2019-01-17 08:58, Tom Herbert wrote:

> to which packets are addressed.

A second rule - per RFC 1812 - is that forwarding devices modify only
the opcount and certain IP options. 

> Intermediate nodes performing host
> violated rule #1.

They consume those packets, they are acting as if they think all
outgoing dest addresses are theirs and assigned to their interface.
I.e., from the outside, they act exactly as if their interface has ALL
those addresses. 

> If they're keeping flow state in the network
> in order for state to be maintained.

Agreed. And because they act as host, they fail badly when they're not
deployed under host assumptions and requirements. 

> This is not a problem hosts have,

It is - that's implicit in the function of their reassembly mechanism
and a reason why it's hard to deploy "virtual" hosts that live in
multipe places on the network (e.g., for single-IP load balancing server
farms, for virtual deployment of copies of a single IP in multiple
places, etc.) 

> the same path to a destination.

Path, no - agreed. But *destination*, yes. Again, the host *is* the
destination. 

> Without additional requirements on

Other way around; multipath breaks stateful devices (they don't work as
desired) when they're not deployed *as the hosts they actually are and
always have been* (at least when they perform certain functions we're
talking about there). 

> should be a specification to that effect.

When you act as a host, you have to follow the rules of being a host. 

> It's not enough to simply
> say they have to follow host requirements. 
> 
> They can't and they clearly

I agree they don't; it's clear they CAN (it would be costly). 

They don't impose any new requirements beyond being the host they are,
though. 

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 13:09:59 -0800

On Thu, Jan 17, 2019 at 12:48 PM Joe Touch <[email-masked]> wrote:
> A second rule - per RFC 1812 - is that forwarding devices modify only the opcount and certain IP options.
>
>
>
> They consume those packets, they are acting as if they think all outgoing dest addresses are theirs and assigned to their interface. I.e., from the outside, they act exactly as if their interface has ALL those addresses.
>
>
> Agreed. And because they act as host, they fail badly when they're not deployed under host assumptions and requirements.
>
>
>
> It is - that's implicit in the function of their reassembly mechanism and a reason why it's hard to deploy "virtual" hosts that live in multipe places on the network (e.g., for single-IP load balancing server farms, for virtual deployment of copies of a single IP in multiple places, etc.)
>
>
> Path, no - agreed. But *destination*, yes. Again, the host *is* the destination.
>
>
>
> Other way around; multipath breaks stateful devices (they don't work as desired) when they're not deployed *as the hosts they actually are and always have been* (at least when they perform certain functions we're talking about there).
>
>
>
> They don't impose any new requirements beyond being the host they are, though.
>
Joe,

When they attempt to do host processing on packets that don't belong
to them they're not hosts. And when they do this, they impose a new
requirement that hosts do not have which is that packets for some flow
must consistently the traverse the same intermediate node.

In any case, as I said previously, reassembly is only specified as an
operations at destination hosts. It seems like there might be a need
for in-network reassmbly, or some nodes might be doing it already.
But, in that case we really need the specification of the protocol to
have a meaning discussion about it.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 15:17:08 -0800

> On Jan 17, 2019, at 1:09 PM, Tom Herbert <[email-masked]> wrote:
> 
> Joe,
> 
> to them they're not hosts.

They are every host for whose packets they process.

> And when they do this, they impose a new

It’s not an intermediate node - it’s a host. It’s the same requirement as any host - that a host has one point of attachment to a network for a given IP address.

> operations at destination hosts.

And these are destination hosts because of how they behave, which means they need to reassemble (either actually or logically).

> It seems like there might be a need

It’s not in-network - it’s at a host, but yes, some do reassemble (or its equivalent).

> have a meaning discussion about it.

RFC 791 and 1122 provide everything that is needed.

It’s not new, it’s just not an “intermediate” node. Never was.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 15:19:03 -0800

> On Jan 17, 2019, at 3:17 PM, Joe Touch <[email-masked]> wrote:
> ,,,
> 

FWIW, another way of looking at the problem is this:

- if it was bound to the rules of a host, would it work? Would that be enough to specify it’s constraints?

If so, then maybe that’s further evidence of what it *is*, regardless of what it is sold as.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 17 Jan 2019 15:46:15 -0800

Joe,

You can try this experiment. Provision two stateful firewalls as
egress points of a network. From an internal network network host
connect to an external host of the network. Now changing routing. If
packets to the external network were being routed through firewall A,
changing routing so they go through firewall B. Now take a look at
what happens to packets on the connection. They are dropped at
firewall B because it does not have the connection state that firewall
A has. The connection is no longer viable. You can call these devices
whatever you want, but that doesn't change the fact that stateful
devices in the network, including those that attempt reassembly, break
multi-path.

Tom

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 21:24:20 -0800

When I call them (multihomed) hosts, I never would assume that the experiment you propose would work. However, if I limit the paths to go through only one of those boxes, treating it as the host it is, everything works fine.

That’s why it IS a host. And why I don’t need new rules to understand or explain it.

Joe

> On Jan 17, 2019, at 3:46 PM, Tom Herbert <[email-masked]> wrote:
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 17 Jan 2019 21:24:20 -0800

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 18 Jan 2019 07:39:02 -0800

Joe,

You just stated a new rule. In order for these devices to work we need
to "limit the paths to go through only one of those boxes". If a user
deploying a network doesn't follow this rule they will be in a world
of hurt when things start to fail. Even if someone understands the
rule, consistently routing flows in even a moderately complex network
is a hard problem. This is one reason why transport layer proxies are
popular. They are hosts, addressees of packets, in the network that
terminate transport layer protocols and don't require consistent
routing (but they do entail other complexities).

Tom

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 18 Jan 2019 08:19:59 -0800

> On Jan 18, 2019, at 7:39 AM, Tom Herbert <[email-masked]> wrote:
> 
> to "limit the paths to go through only one of those boxes".

Sticking a phone in a pot of dirt doesn’t make it a houseplant.

Putting these devices where hosts don’t work doesn’t make them routers or anything else.

The rules are simple. If it is a host , it must be run like a host.

If the only way it runs correctly is to act like a host, it is a host.

The Internet doesn’t have a rule that says you can do anything you want. Play by the rules or the system doesn’t work.

> If a user
> of hurt when things start to fail.

Exactly like sticking the phone in a pot of dirt ruins the phone.

> Even if someone understands the
> is a hard problem.

Yes, when you keep sticking phones in pots of dirt, it’s hard to use the phones.

> This is one reason why transport layer proxies are

Non-transparent proxies force users to treat the boxes as hosts, and magically everything starts working again.  Go figure.

Joe