Subject: Re: [Ietf-behave] Requiring NATs to process Out-of-order fragmented TCP packets

From: Pyda Srisuresh <[email-masked]>
Date: Mon, 7 Mar 2005 15:57:31 -0800 (PST)

--- Spencer Dawkins <spencer@xxxxxxxxxxxxx> wrote:
> My understanding from this morning's meeting was that we were leaning 
> toward not requiring Behave-conformant NATs to process out-of-order 
> TCP fragments, with the expectation that endpoints would be doing Path 
> MTU discovery anyway, so fragmentation shouldn't be an issue.
> 
[suresh] If you are talking about processing out of order IP fragments in
general, I believe, the chairs said we would take this to the list. If NATs do
not handle IP fragments correctly, it is a major problem for applications that
use IP fragments (ex: NFS over UDP). I believe, this is a MUST for NATs.

regards,
suresh

> I ran into Mark Allman this afternoon and asked him if he had an 
> opinion on the robustness of Path MTU discovery on the Internet. He 
> does, and has recently co-authored a paper that discusses this 
> (available at http://www.icir.org/mallman/papers/tcp-evo-submit2.ps). 
> In section 4.3 of the paper, he's seeing that about one-third of the 
> 85,000 servers identified by the NLANR Web Caching project don't 
> attempt Path MTU discovery at all, and another 17 percent attempt it 
> and fail. The failure rate is even higher for a list of 500 popular 
> web server (35%).
> 
> So I'm wondering if we need to think about this a little more?
> 
> Spencer
> 
> p.s. I copied Mark, just in case I'm misunderstanding the paper... 
> 
> 
> _______________________________________________
> Ietf-behave mailing list
> Ietf-behave@xxxxxxxxxxxxxxxxxxx
> https://list.sipfoundry.org/mailman/listinfo/ietf-behave
>

---

From: Cullen Jennings <[email-masked]>
Date: Tue, 15 Mar 2005 17:09:35 -0800 (PST)

Not as chair 

I look forward to getting the TCP folks input on this but I believe one of
the issues in trying to say that devices should process out of order packets
is what to do about the DOS attacks this creates if lots of out of order
packets are sent. 

Right now I don't see how we could recommend that a NAT process out of order
packets because we would need to say that if you got lots of them, then the
NAT would need to stop doing the out of order processing or else it would
run out of memory. This would be exactly the sort of non deterministic
behavior that we are trying to stop.

Cullen

---

From: Kaushik Biswas <[email-masked]>
Date: Tue, 15 Mar 2005 21:45:37 -0800 (PST)

Valid point. In my opinion we should be putting a MAY against this 
requirement in the TCP draft on the same lines as UDP draft has. In the 
Security Section there is discussion cautioning NAT implementers about 
this. This might need to be expanded.

Kaushik

At 05:09 PM 3/15/2005 -0800, Cullen Jennings wrote:

Not as chair

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 16 Mar 2005 08:51:50 -0800 (PST)

I think much the UDP discussion led to the idea that the MAY in the document
are not very useful. We are trying to describe a set of characteristics on
how a NAT behaves that allows applications to make assumptions that they can
reply on. The MAY don't help this in many cases. I agree with you that  this
is the sort of stuff where we want to provide advice and cautions to the NAT
implementers that warn them about the consequences of certain
implementations. 

Cullen (not as chair)

---

From: Senthil Sivakumar <[email-masked]>
Date: Wed, 16 Mar 2005 15:11:41 -0800 (PST)

Not as chair

I look forward to getting the TCP folks input on this but I believe one of
the issues in trying to say that devices should process out of order packets
is what to do about the DOS attacks this creates if lots of out of order
packets are sent.
Right now I don't see how we could recommend that a NAT process out of order
packets because we would need to say that if you got lots of them, then the
NAT would need to stop doing the out of order processing or else it would
run out of memory. This would be exactly the sort of non deterministic
behavior that we are trying to stop.
Conversely, if you don't recommend NATs to process out of order fragments,
some NATs will store the fragments until the first fragment is received, 
some might
drop them,  some might just forward them (if they can). This causes 
non-deterministic
behavior as well.

Now the solution for preventing NATs from DoS attacks could be that there 
is a threshold
for the number of fragments that can be queued and the time it can be 
queued. I believe
these thresholds will vary based on the memory, CPU and network latency.

Senthil

Cullen

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 16 Mar 2005 16:16:36 -0800 (PST)

Sounds like we are saying about the same thing. If the amount of memory is
set to 0 then one thing happens, if it is set above that, then something
else happens. I think the end story for the applications is going to be they
can't make assumptions about what will happen to out of order fragments.

---

From: Dan Wing <[email-masked]>
Date: Wed, 16 Mar 2005 16:48:18 -0800 (PST)

On Mar 16, 2005, at 4:16 PM, Cullen Jennings wrote:

Sounds like we are saying about the same thing. If the amount of 
memory is
set to 0 then one thing happens, if it is set above that, then 
something
else happens. I think the end story for the applications is going to 
be they
can't make assumptions about what will happen to out of order 
fragments.
Which is a nice way of saying "if fragmentation occurs, everything could
break."
This makes it sounds like applications need to prevent the network from
fragmenting packets, then, which means they MUST do PMTUD for both UDP
and for TCP?
-d

---

From: Senthil Sivakumar <[email-masked]>
Date: Wed, 16 Mar 2005 16:58:46 -0800 (PST)

The end application may not see the complete data if NAT
drops or don't know how to process fragments. So either
you have to make NATs support fragments and/or the end
hosts do PMTU discovery to ensure no fragmentation occurs
in the network. I would say you have to do both. The recommendation
should be NATs should support handling fragments but at the same time
protect itself from attacks.
Senthil

At 04:16 PM 3/16/2005 -0800, Cullen Jennings wrote:

---

From: Kaushik Biswas <[email-masked]>
Date: Wed, 16 Mar 2005 16:59:52 -0800 (PST)

At 04:48 PM 3/16/2005 -0800, Dan Wing wrote:

Sounds like we are saying about the same thing. If the amount of memory is
set to 0 then one thing happens, if it is set above that, then something
else happens. I think the end story for the applications is going to be they
can't make assumptions about what will happen to out of order fragments.
Which is a nice way of saying "if fragmentation occurs, everything could
break."
This makes it sounds like applications need to prevent the network from
fragmenting packets, then, which means they MUST do PMTUD for both UDP
and for TCP?
And a good chunk of the ones don't do PMTUD. Spencer had fwd'd a paper for 
the same to the alias. Being a box-in-middle NAT is most likely to get out 
of order fragments.

Kaushik

-d

---

From: nagendra modadugu <[email-masked]>
Date: Wed, 16 Mar 2005 20:09:47 -0800 (PST)

> fragmenting packets, then, which means they MUST do PMTUD for both UDP

I'm for processing of in-order fragments--from what I discovered while 
working on DTLS, PMTUD is rarely turned on, and some OS's don't even
correctly implement getsockopt() to return the MTU value received in an
ICMP MTU exceeded message.  This makes it hard for even MTU-mindful
applications to do the right thing.

Out-of-order fragments is a wash--it doesn't seem to be worth the effort
(even though an DoS attack against a buffering-NAT isn't particularly
effective).  

Discussion below.

Out-of-order Fragments:

Suppose buffering of fragments is supported by the NAT, so that IP
packets can be correctly reconstructed and forwarded.  An attacker could 
do the following:

  (1) Generate an IP packet with SA={IP of some popular server, e.g.
      some IMAP server}, DA={external IP address of the NAT}.
  (2) Fragment the IP packet into F1, F2, .., Fn, such that the TCP
      header lies entirely in F1.
  (3) Transmit F2, F3, .., Fn (skipping F1).
  (4) The NAT buffers all the fragments received since, the (SA,SD) pair
      is likely to be a valid tuple for current connections, and the NAT
      has no further information upon which to filter the fragments
      (typically port numbers).

This is potentially a bad situation--an attacker can force the NAT to buffer 
packets (a) without having to be on a connection path, and (b) can be 
successful with a very high probability.

The way to counter such an attack is to timeout fragment buffering.
One way to implement this is for the NAT to fix a time window in which
all the fragments for an IP packet must arrive--to be generous, say 100ms.  
So if the attacker can attack at line rate, say 10Mbps, then the NAT 
needs to buffer at most ~125kB of data (100ms worth of data).

It's possible that the numbers I've picked are way off, but if they're
anywhere close to realistic, the attack isn't terribly effective--it doesn't 
take much memory on part of the NAT to implement out-of-order fragment 
processing.  Besides which, attacking at line rate is indicative of a
much bigger problem.

I recall hearing a story about a Juniper 'bug' in their routers that
resulted in reordering packets.  Apparently, this bug had caused some
software to break, eventually forcing Juniper to fix the bug. 
If any of this story is true, then I'm leaning towards concluding that
out-of-order packets are a rareity (if packet N+1 is received before
packet N, then chances are that packet N has been dropped along the
way) in which case out-of-order fragment processing is not much use
anyway.

Of course, things could go terribly wrong if the timer code is buggy,
or timer values are too large--which is not an unlikely scenario.

In-order-fragments:

In-order fragments can be easily processed without need for buffering 
based on their IP identifier fields.  Since there is no buffering, the
worst an attacker can do is slip packets through the NAT from the outside 
(an attack that can also be accomplished without employing IP fragments).