Subject: [v6ops] KSK Rollover and IPv6 Fragmentation

From: Fred Baker <[email-masked]>
Date: Tue, 14 Mar 2017 10:05:33 +0100

Many of us likely know that the DNS Root Key will be changing in October. If you don't know that, you know it now. If you want a view into the testing that is being done related to that, take a look at http://go.icann.org/KSKtest. I bring it up in this forum primarily because it might affect services being offered by operators in this working group, and they might want to do similar testing prior to the event.

But we might collectively look through https://blog.apnic.net/2016/11/15/scoring-dns-root-server-system/, which comments on the rollover, in the context of another thread that has come and gone several times here - "could we simply deprecate IPv6 fragmentation? Please?" The answer is something along the lines of "sure, fine, unless you want the DNS Key Rollover to go smoothly". Yes, DNS root servers could force a fall-back to TCP for 20 days. We would be testing some code in a few places that isn't usually tested - and requiring that it work. When Mirai tipped parts of Dyn over, it made the papers. Forcing certain DNS exchanges to shift from UDP to TCP for 20 days - well, let's just say "I would worry".

Food for thought.

---

From: Philip Homburg <[email-masked]>
Date: Tue, 14 Mar 2017 12:42:30 +0100

>But we might collectively look through https://blog.apnic.net/2016/11/15/scori
>ng-dns-root-server-system/, which comments on the rollover, in the context of 
>another thread that has come and gone several times here - "could we simply de
>precate IPv6 fragmentation? Please?" The answer is something along the lines o
>f "sure, fine, unless you want the DNS Key Rollover to go smoothly". Yes, DNS 
>root servers could force a fall-back to TCP for 20 days. We would be testing s
>ome code in a few places that isn't usually tested - and requiring that it wor
>k. When Mirai tipped parts of Dyn over, it made the papers. Forcing certain DN
>S exchanges to shift from UDP to TCP for 20 days - well, let's just say "I wou
>ld worry".

ICANN has a nice technical analysis that I can never find.

In any case, for DNS over UDP, we cannot assume that a root DNS server will
do PMTU discovery. The code will be there, but it just doesn't work.

So there are three options:
1) Fragment at 1500. For the KSK rollover that means no fragmentation
2) Fragment at 1280. For the KSK rollover that means that fragmentation will
   happen for a DNSKEY query.
3) Truncate at 1280. For the KSK rollover that means that truncation will
   happen for a DNSKEY query.

Assuming that option 3) is out (increase in TCP traffic is too risky) then
2) is technically correct. However, there is wide spread filtering of
fragments. So 2) results in a significant loss rate. 

So in practice, 1) minimizes loss but is bad for everybody behind a link with 
a low MTU.

In practice, all of this doesn't really matter because the number of
IPv6-only recursive resolvers is very low. And there is no issue with IPv4.

One thing that makes me sad, is that it should be relatively easy to
implement PMTU blackhole detection on the DNS client side (set the EDNS 
udp size to what fits in 1280 after a few timeouts). But just like the PMTU
blockhole detection for TCP, that doesn't seem wide spread.

---

From: Mark Smith <[email-masked]>
Date: Tue, 14 Mar 2017 23:18:48 +1100

> But we might collectively look through https://blog.apnic.net/2016/11/15/scoring-dns-root-server-system/, which comments on the rollover, in the context of another thread that has come and gone sev…

I assume that people drop fragments because there are no port numbers
to look at/filter on in them. Shifting fragmentation into the UDP
layer would overcome that issue. There may be some light at the end of
the tunnel.

"Transport Options for UDP"
https://tools.ietf.org/html/draft-touch-tsvwg-udp-options-05

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Mark Andrews <[email-masked]>
Date: Wed, 15 Mar 2017 00:09:05 +1100

>    happen for a DNSKEY query.

4) fragment at 1280 if DNS message size > 1432

This reduces the impact of firewalls that drop packet fragments.
This reduces the impact of the firewalls that drop PTB or routers that
don't reliable generate PTB.
 
> blockhole detection for TCP, that doesn't seem wide spread.

Blackhole detection has been implemented for a long time.

The hard part with DNS is that non answers can be due to:

* dead server
* packet loss
* PMTU discovery in progress
* EDNS being blocked/dropped (or only the first EDNS message is responded to,
			      older Microsoft servers have this behaviour)
* EDNS OPTIONS being filtered
* fragmentation filtering

Then even if the PTB gets through, the DNS authoritative server can't
construct a new response as the OPT record is at the end of the
message and its contents are crucial to regenerating the response.
That is usually missing from the PTB ICMP contents.

The DNS recursive server is trying to workout what is the cause and
return a response to the initial client all in < 3 seconds.

draft-touch-tsvwg-udp-options could help here but it is a long term
solution.

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: [email-masked]
Date: Tue, 14 Mar 2017 14:12:39 +0100

Mark,

> This reduces the impact of firewalls that drop packet fragments.

How so? Wouldn't that still lead to one fragment without the transport header?

> don't reliable generate PTB.

Routers never reliably generate PTB. That's in the spec.

> draft-touch-tsvwg-udp-options could help here but it is a long term

Cheers,
Ole

---

From: Mark Andrews <[email-masked]>
Date: Wed, 15 Mar 2017 00:34:39 +1100

In message <[email-masked]>, otroan@employe
es.org writes:
>
> Mark,
>
> >>> But we might collectively look through
> https://blog.apnic.net/2016/11/15/sco
> >> ri
> >>> ng-dns-root-server-system/, which comments on the rollover, in the
> context o
> >> f
> >>> another thread that has come and gone several times here - "could we
> simply
> >> de
> >>> precate IPv6 fragmentation? Please?" The answer is something along
> the lines
> >> o
> >>> f "sure, fine, unless you want the DNS Key Rollover to go smoothly".
> Yes, DN
> >> S
> >>> root servers could force a fall-back to TCP for 20 days. We would be
> testing
> >> s
> >>> ome code in a few places that isn't usually tested - and requiring
> that it w
> >> or
> >>> k. When Mirai tipped parts of Dyn over, it made the papers. Forcing
> certain
> >> DN
> >>> S exchanges to shift from UDP to TCP for 20 days - well, let's just
> say "I w
> >> ou
> >>> ld worry".
> >>

It lets DNS messages between 1280 and 1432 be emitted without being
fragmented and removes most PMTUD issues around IPv6 in IPv4.  It's
a balancing act that should have to be done.

Firewall vendors could also be smarted about how they allow reply
traffic through.  There really is no reason to block all fragments.

> Routers never reliably generate PTB. That's in the spec.

But there is reliable enough and ridiculously rate limited.

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 Mar 2017 11:38:45 -0700

>    happen for a DNSKEY query.

IMO, if things are going to break, they ought to break because others
fail to support existing specs, not the DNS operators.

Thus, IMO, fragment at MIN(path MTU if known, 1280), truncate at 1500
(as per RFC2460).

Joe

---

From: Mark Andrews <[email-masked]>
Date: Wed, 15 Mar 2017 09:40:04 +1100

> Thus, IMO, fragment at MIN(path MTU if known, 1280), truncate at 1500

It is fragment at 1280 and truncate at EDNS UDP buffer size.  PMTUD
is a right royal pain for DNS servers even for TCP.  When we actually
did this we get bad press because we are generating responses that
get blocked by stupid firewalls.

If you let through UDP fragments, offset != 0 and test the ports
on UDP fragments with offset == 0 the worst you have is some extra
traffic and fragment reassembly failure counters going up.  If you
add source and destination address checks with a resetable timer
each time a new outgoing UDP packet that matches the address pair
to the first half above you remove most of the extra traffic.

Unfortunately I don't believe you can buy firewall that does this
simple to implement strategy that lets fragmented responses work.
Firewall vendors go for the nuclear option.  Block all fragments.

Of that list of cause for non responses, firewall cause about half
of them:

In the future there wall also be operational issues with

* EDNS version != 0 being filtered
* EDNS flag XXXX being filtered

At the moment the last two only impact testing traffic and protocol
developers needing to decide how to how to fix / extend the DNS.
They have already stopped DNSOP bumping the EDNS version number
when cleaning up EDNS unknown option handling in RFC 6891.  EDNS(0)
undefined => EDNS(1) ignore.  This is how it should have been done
in a sane world without stupid firewall vendors that just block
anything that is not what is currently being used irrespective of
whether there was well defined behaviour for how to handle EDNS(1)
queries when you only support EDNS(0) or not.

Mark

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 Mar 2017 15:48:46 -0700

On 3/14/2017 3:40 PM, Mark Andrews wrote:
>> ...
> It is fragment at 1280 and truncate at EDNS UDP buffer size. 

Fragment at 1280, because that's the min guaranteed by IPv6 as a hop MTU
(if you know better via PMTUD, use that).

Truncate at 1500 because that's the min guaranteed reassembly by IPv6.

UDP has nothing to do with it, unless you're using some sort of UDP
reassembly (see draft-touch-udp-options). Otherwise, it's the IP EMTU_R
that matters.

> PMTUD
> If you let through UDP fragments, 
You mean IP fragments... ;-)
> offset != 0 and test the ports
Understood, and that means their products do not support IPv6. Until we
actually publish that list, we can't do much.

However, I really don't want to make products that don't support
near-1500b links "non compliant" with IPv6, e.g., by assuming an IP
fragmentation boundary that's larger. If something is going to be
flogged for breaking, let it be the broken one.

The only way around the fragment blocking issue is to fragment at a
higher layer - above UDP, or (per the draft above), within UDP.

Joe

---

From: Mark Andrews <[email-masked]>
Date: Wed, 15 Mar 2017 10:13:12 +1100

> Truncate at 1500 because that's the min guaranteed reassembly by IPv6.

EDNS UDP buffer size is supposed to be set at less than or equal
reassembly size supported by the node.  1500 is the minimum that
needs to be supported by all nodes.  DNS can negotiate bigger sizes
and that is usually around 4K.

For TCP the segment should fit in a 1280 packet.

Setting IPV6_USE_MIN_MTU to 1 should get both UDP fragmentation and
TCP MSS correct unless there is a bug in the IPv6 stack.

The UDP packet size is controlled by the advertised EDNS UDP buffer size
option.

> You mean IP fragments... ;-)

Well IPv6 fragments with a final next header of UDP.  TCP shouldn't
be sending fragmented IPv6 packets unless the sending stack is
broken.  TCP is supposed to consider the actual MTU's in use.

If draft-touch-udp-options become reality I will need to be more
precise. :-)

> higher layer - above UDP, or (per the draft above), within UDP.

Which still requires firewalls to be fixed.  They will need to allow
more than 1 response packet to the DNS/UDP request.

Mark

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 Mar 2017 16:16:19 -0700

> The UDP packet size is controlled by the advertised EDNS UDP buffer size

Ahh - thanks. Yes, if you know that the EMTU_R is larger (which is what
the EDNS UDP buffer size would indicate), then indicate that.

> more than 1 response packet to the DNS/UDP request.
Yes, and they should already ;-)

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Mar 2017 11:16:54 -0700

On 3/14/2017 5:18 AM, Mark Smith wrote:
> to look at/filter on in them.

Some people also filter fragments to avoid reassembly attacks.

> Shifting fragmentation into the UDP
No argument here ;-)

Though the "light at the end of the tunnel" is covered here ;-p

https://tools.ietf.org/html/draft-ietf-intarea-tunnels

Joe

---

From: [email-masked]
Date: Thu, 16 Mar 2017 10:18:30 +0100

Mark,

Thanks for a great summary!

> queries when you only support EDNS(0) or not.

To talk about IPv4 for a second (I know, I know). It isn't only firewall vendors.
In all the IPv4 sharing mechanisms IPv4 fragments is a royal pain to deal with, and very costly.
And we can agree that those mechanisms are evil, but we are were we are.

I have implemented virtual reassembly in VPP for MAP-E. If fragments arrive in order and are relatively closely spaced in time and packet flow, it's fine, but as soon as they are out of order, which require buffering it very quickly appears like a DOS attack.
Meaning fragments will regardless have a much higher drop probability. In addition comes the issues of sharing the IP identifier space among multiple users.

I tried to get some data on IPv4 fragmentation but that was hard to find. In the packet traces I found most of the fragmented traffic was either DOS attacks, stuff that didn't reassemble, or DNS attacks. If anyone has better data let me know.

I'm not quite convinced that a protocol without a 3 way handshake is a good idea anymore.

Cheers,
Ole

---

From: Mark Andrews <[email-masked]>
Date: Thu, 16 Mar 2017 22:57:42 +1100

In message <[email-masked]>, otroan@employe
es.org writes:
> I have implemented virtual reassembly in VPP for MAP-E. If fragments
> arrive in order and are relatively closely spaced in time and packet
> flow, it's fine, but as soon as they are out of order, which require
> buffering it very quickly appears like a DOS attack.
> Meaning fragments will regardless have a much higher drop probability. In
> addition comes the issues of sharing the IP identifier space among
> multiple users.
>
> I tried to get some data on IPv4 fragmentation but that was hard to find.
> In the packet traces I found most of the fragmented traffic was either
> DOS attacks, stuff that didn't reassemble, or DNS attacks. If anyone has
> better data let me know.
>

There is also a lot of legitimate fragmented DNS traffic and it is growing
as more zones get signed.  Almost all of it gets through provided you don't
have a local firewall blocking it.

[rock:~/git/bind9] marka% netstat -s | grep frag
	35608 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	45014 fragments received
	1 fragment dropped (dup or out of space)
	84 fragments dropped after timeout
	0 fragments that exceeded limit
	1 output datagram fragmented
	2 fragments created
	0 datagrams that can't be fragmented
		fragment: 45014
[rock:~/git/bind9] marka% uptime
22:53  up 23 days, 12:12, 8 users, load averages: 1.35 1.07 0.71
[rock:~/git/bind9]

Mark

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Mar 2017 07:28:04 -0700

On 3/16/2017 2:18 AM, [email-masked] wrote:
> And we can agree that those mechanisms are evil, 
Are you saying that IPv4 fragments are worse than IPv6?

IP source fragmentation is *required * as long as we use 'bare' tunnels
(IP over IP with no intervening layers). Intervening layers only move
reassembly to a different layer.
...
> I have implemented virtual reassembly in VPP for MAP-E. If fragments arrive in order and are relatively closely spaced in time and packet flow, it's fine, but as soon as they are out of order, whic…
Everything is a DOS attack. There was a time when IP header processing
couldn't keep pace (thus the need for label switching that evolved into
MPLS), where even IP itself was basically a DOS attack.

...
> I'm not quite convinced that a protocol without a 3 way handshake is a good idea anymore.
But that handshake is itself a significant opportunity for a DOS attack.

Ultimately, any message you don't want can be considered an attack.
However, I'm more concerned about the constant attack we endure from
vendors trying to underdesign protocol implementations.

Joe

---

From: [email-masked]
Date: Thu, 16 Mar 2017 15:47:28 +0100

Joe,

> Are you saying that IPv4 fragments are worse than IPv6?

Yes, but because the IPv4 header is now 28 bytes and the network is forced to do routing on address + port.
That doesn't apply to IPv6.

> vendors trying to underdesign protocol implementations.

Cheap shots at vendors aside, the code is open source, feel free to improve it.
https://git.fd.io/vpp/tree/src/vnet/map/ip4_map.c

But, any time a packet / packet train has a significantly higher processing cost than others, drop probability raises.
Regardless it is not within the bounds of physics to follow RFC791 recommendations for reassembly.
At 100Mpps I don't have 15 seconds * 100M buffer space. ;-)
Just accept that IPv4 fragments are unlikely to have a happy life as a consequence of IPv4 address run out. Tough luck.

Best regards,
Ole

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Mar 2017 08:02:54 -0700

> Yes, but because the IPv4 header is now 28 bytes and the network is forced to do routing on address + port.

The network is never *forced* to route on anything beyond IP destination
address.

> That doesn't apply to IPv6.

Perhaps we can take the clarification off-list; I'm still not
understanding your point. In both cases, all but the first fragment lack
a port. I *wish* the flow-ID were a sufficient replacement, but AFAICT
there are still routers that rely on port inspection even when the flow
ID is nonzero (incorrectly, IMO).

...
> At 100Mpps I don't have 15 seconds * 100M buffer space. ;-)Just accept that IPv4 fragments are unlikely to have a happy life as a consequence of IPv4 address run out. Tough luck.
>
That's why we relaxed the IPv4 ID uniqueness requirements to track
expected reordering in RFC 6864, which updates RFC791.

Joe

---

From: [email-masked]
Date: Thu, 16 Mar 2017 16:12:41 +0100

Joe,

> The network is never *forced* to route on anything beyond IP destination

Perhaps this is where the misunderstanding is.
The network is _forced_ to route on address + port. To make the IPv4 Internet continue to operate there is no choice other than sharing an IPv4 address among more and more end users.

Cheers,
Ole

---

From: Owen DeLong <[email-masked]>
Date: Thu, 16 Mar 2017 10:22:22 -0700

> On Mar 16, 2017, at 8:12 AM, [email-masked] wrote:
> 
> Joe,
> 

The network is not forced to route on address + port.

Some operators choose to continue to support a protocol that should have been long since deprecated rather than implementing IPv6. The consequence of that choice is a self-inflicted wound.

Owen

---

From: [email-masked]
Date: Thu, 16 Mar 2017 18:32:46 +0100

> Some operators choose to continue to support a protocol that should have been long since deprecated rather than implementing IPv6. The consequence of that choice is a self-inflicted wound.

I do like your fundamentalist stance.

Ole

---

From: Mark Andrews <[email-masked]>
Date: Fri, 17 Mar 2017 07:35:49 +1100

In message <[email-masked]>, otroan@employe
es.org writes:
> The network is _forced_ to route on address + port. To make the IPv4
> Internet continue to operate there is no choice other than sharing an
> IPv4 address among more and more end users.

Not really. You can still route on addresses only provide you have
enough addresses that you can have a unique <src',dst> tuples for
each <src,dst> from the inside of the NAT.  The problem is that we
are used to the case with homes where you only have a single source
address to play with so that only leaves ports.  For CGN's we have
a pool of addresses to pick the NAT's source address from.

Mark

> >> But, any time a packet / packet train has a significantly higher
> processing cost than others, drop probability raises.
> >> At 100Mpps I don't have 15 seconds * 100M buffer space. ;-)Just accept
> that IPv4 fragments are unlikely to have a happy life as a consequence of
> IPv4 address run out. Tough luck.
> >>

---

From: [email-masked]
Date: Thu, 16 Mar 2017 22:41:20 +0100

Mark,

> a pool of addresses to pick the NAT's source address from.

Not doing CGN. Stateless forwarding. RFC7596/97,99.

Ole

---

From: Mark Andrews <[email-masked]>
Date: Fri, 17 Mar 2017 10:40:05 +1100

> Not doing CGN. Stateless forwarding. RFC7596/97,99.

An engineering trade off.  <src,dst> tuple state vs needing to see
ports.  In both cases its keeping a protocol alive that should have
died in 1995 because the world had effectively run out of IPv4
addresses.

Mark

---

From: Simon Hobson <[email-masked]>
Date: Fri, 17 Mar 2017 16:22:41 +0000

> I do like your fundamentalist stance.

I've often commented to people that if all the effort that went into "working around" the problems caused by deliberately breaking IPv4 had gone into developing & adopting IPv6 then we'd not be having these discussions ! The only reason so many people see nothing wrong with NAT is because they are lucky enough to only be using protocols/services where someone else has "fixed" the problems.

In the last two days (at work) I've had two issues directly caused by broken networks (aka NAT) - both with SIP, completely different customers/setups.