Subject: Re: [Int-area] New Version Notification for draft-bonica-intarea-gre-mtu-00.txt

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 01:17:38 -0700

Hi, Fred,

On 6/26/2013 8:42 AM, Templin, Fred L wrote:
> Hi Joe,
>
>> -----Original Message-----
>> From: Joe Touch [mailto:[email-masked]]
>> Sent: Tuesday, June 25, 2013 8:10 PM
>> To: Templin, Fred L
>> Cc: Carlos Pignataro (cpignata); Ronald Bonica; Internet Area
>> Subject: Re: [Int-area] New Version Notification for draft-bonica-
>> intarea-gre-mtu-00.txt
>>
>> Hi, Fred,
>>
>> On 6/25/2013 3:32 PM, Templin, Fred L wrote:
>>> Hi Joe,
>>>
>>> The constraint you seem to be missing is that RFC2460 only requires
>>> IPv6 nodes to reassemble as much as 1500 bytes.
>>
>> Right; sorry for not noticing that further. Had IPv4 on the brain.
>
> OK, but IPv4 also has a limit of minMRU=576. So, we have:
>
>    IPv4 minMTU = 576 (*)
>    IPv4 minMRU = 576
>    IPv6 minMTU = 1280
>    IPv6 minMRU = 1500
>
> (*) Even though the specs say that IPv4 minMTU = 68, everyone
> seems to be saying that for practical purposes it is now 576.

There needs to be a difference between the minMTU and the minMRU; if 
not, then IP-in-IP tunnels will never succeed without a separate 
fragmentation and reassembly layer - and although SEAL provides that, we 
currently do not require anything like that for X-in-X encapsulation.

>> ...
>>> Granted, my algorithm is asking GRE egress tunnel endpoints to
>>> configure a reassembly buffer size of 1500 *plus* the tunnel
>>> encapsulation overhead, which is more than RFC2460 ensures.
>>
>> Agreed.
>
> Right. So, then the fragmentation threshold for GRE over IPv6
> needs to be (1500 - HLEN) instead of 1500.
>
>> ...
>>> Also think about it this way. Why should the tunnel work hard to
>>> pass a packet that is larger than 1500 bytes and also larger than
>>> the minimum link MTU within the tunneled path when the source could
>>> instead adjust the size of the packets it is sending downward?
>>
>> That's not a good argument. I agree that all that need be satisfied is
>> the largest reassembled IPv6 packet, but not that we should assume the
>> source can or will adjust downward - if that were the case, you could
>> ignore the overhead of GRE and just claim that the MTU is 1280.
>
> In my lab, my implementation sends a single packet for packet
> sizes no larger than (1280 - HLEN). For packets that are larger
> than (1280 - HLEN) but no larger than 1500, it fragments the
> packet into two pieces and sends both pieces to the tunnel far
> end which then needs to reassemble. But, reassembly will be
> delayed by the inter-packet delay plus the time it takes to
> run the reassembly routine, and the delay is measurable. So,
> we really only want to use frag/reass when we absolutely have to.

My view is that discovering the path MTU for the purposes of avoiding 
frag/reassembly is either necessary or an optimization (such as you 
mention above, to avoid delays and conserve resources). That's in 
keeping with the "work first, optimize later" approach of the Internet.

So frag/reassemble unless you CANNOT. Send too-big messages back to help 
the system optimize. But never deliberately drop anything you could have 
sent with frag/reassembly.

Joe

>
> Thanks - Fred
> [email-masked]
>
>> Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 01:27:07 -0700

On 6/26/2013 1:09 PM, Linda Dunbar wrote:
> Ron,
>
> Those applications ignore the ICMP messages and keep sending data
> frames as if there is no ICMP message received.
> We ran into this problem in testing Video distribution system. After
> the first few frames, the receivers can't receive anything. After
> lengthy debugging, we found out that an intermediate node drops packets
> when GRE encapsulated frames exceed the link MTU. We tested that ICMP
> message is sent back, but the source node didn't respond. We tried
> several setting as Joe pointed out. Some guest OSs change behavior, some
> don't.

Who to fix in this situation depends on how the ICMPs received are being 
handled.

Are they being sent to the transport layer?

Is the app tracking the errors?

It might or might not be the OS's fault.

Joe

> Linda
>> From: Ronald Bonica [mailto:[email-masked]]
>> Sent: Wednesday, June 26, 2013 8:00 AM
>> To: Linda Dunbar; Joe Touch; Carlos Pignataro (cpignata)
>> Cc: Internet Area
>> Subject: RE: [Int-area] New Version Notification for draft-bonica-
>> intarea-gre-mtu-00.txt
>>
>> Linda,
>>
>> What do these platforms do when they receive an ICMP Packet too big
>> from a router that isn't a tunnel ingress?
>>
>>                                              Ron
>>
>>
>>> From: Linda Dunbar [mailto:[email-masked]]
>>> Sent: Tuesday, June 25, 2013 10:05 AM
>>> To: Joe Touch; Carlos Pignataro (cpignata)
>>> Cc: Ronald Bonica; Internet Area
>>> Subject: RE: [Int-area] New Version Notification for draft-bonica-
>>> intarea-gre-mtu-00.txt
>>>
>>> Ron,
>>>
>>> Your draft recommends that the ingress node discards the frame and
>>> sends ICMP msg to the source node when the size of GRE encapsulated
>>> frame exceeds link MTU.
>>> We experienced that Window XP doesn't adjust frame size after
>> receiving
>>> ICMP message. Many Linux based applications don't do anything either
>>> with the MTU ICMP message. I think your draft should point those
>> issues
>>> out and recommend fragmentation approach under those circumstances.
>>>
>>> Linda
>>>
>>>
>>>
>>
>>
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 15:34:47 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 1:18 AM
> To: Templin, Fred L

With IPv4, there is no difference between minMTU and minMRU.
Tunnels over IPv4 therefore set DF=0 to allow for in the
network fragmentation if necessary.

With IPv6, minMTU is smaller than minMRU but that does not
guarantee that a packet sent by the ingress can be received
by the egress without fragmentation. RFC2473 acknowledges
this by using fragmentation at the ingress as a limiting
condition for when the MTU within the tunnel becomes too
small. This can happen for example if there is a 1280 MTU
link within the tunnel, if there are nested encapsulations
within the tunnel, etc.
 
> sent with frag/reassembly.

What the SEAL approach fosters is a natural transition to larger
packets where packets up to 1500 are accommodated using fragmentation
if necessary but larger packets are allowed through in one piece as
long as they fit the available MTU. Then, hosts can reasonably assume
that their 1500 packets will get through, but to send anything larger
than that they should use RFC4821. The hosts will notice that they
get better performance using larger packets, and we will be well on
our way to an 8KB Internet.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 13:05:10 -0700

Hi, Fred,

> network fragmentation if necessary.

But then that's useless. Let's say you already send just 576, and set 
DF=1, and that packet encounters an IPv4 tunnel. You add 20 bytes of 
header, resulting in 596.

At the tunnel ingress, you can't fragment the inner packet because DF=1 
- and why shouldn't it be set? You're using the minMTU.

At that ingress, you can't fragment the outer packet because you would 
need the egress to reassemble something that is 596 -- larger than the 
egress ever expected to reassembly (minMRU).

So you drop the packet and send an ICMP too-big back to the source, who 
drops it because they're already sending minMTU packets and doesn't 
think it should have to drop the MTU below that.

AFAICT, you now have broken the path completely.

> by the egress without fragmentation.

No, but it does guarantee that the packet can traverse a tunnel and 
still make it to its destination.

The difference between minMTU and minMRU is the amount of accumulated 
headers you can accommodate by tunneling. At 1500-1280, that's 5 levels 
of nested IPv6 tunnels, or more than a few IPsec tunnel-mode tunnels if 
needed.

That's not as much as I'd like, but it's at least non-zero.

 > RFC2473 acknowledges

Yes, but if minMRU == minMTU, then the number of encapsulations 
supported is zero.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 20:27:59 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 1:05 PM
> To: Templin, Fred L
> >> fragmentation and reassembly layer - and although SEAL provides
> that,
> >> we

For IPv6-within-IPv4 at least, most of these "transition mechanism"
tunnels assume a minMRU of 1500 even though the IPv4 specs say that
the MRU is 576. Sure, that is broken but it seems to have become the
IPv6 transition mechanism "best practice".

> still make it to its destination.

Right, but remember that the IPv6 minMRU is only 1500 bytes.
So, we can't assume an unlimited reassembly buffer on the
tunnel egress.

> of nested IPv6 tunnels, or more than a few IPsec tunnel-mode tunnels if

I addressed this point on the IPv6 list, but think for a moment
what this means. It means that the fist tunnel ingress would have
to configure a 1280 MTU, the second tunnel ingress would need to
configure a 1320 MTU (to accept encapsulated packets from the first),
the third tunnel ingress would need to configure a 1360 MTU (to
accept encapsulated packets from the second), etc. up to 5 levels
of nesting. I don't know about you, but I can imagine many situations
where it is not possible for a single operator to lay hands on every
tunnel ingress so as to carefully set each MTU in this way (I gave
one example on the IPv6 list).

> That's not as much as I'd like, but it's at least non-zero.

I think the maximum acceptable level of nesting is somewhere
between 5 and 10 before the nesting would be declared "recursive"
(i.e., a routing loop). SEAL allows up to 8 levels of nesting.

> Yes, but if minMRU == minMTU, then the number of encapsulations

Right. That's why IPv6-in-IPv4 transition mechanisms cheat and
assume 1500 when the specs say they can only assume 576.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 13:50:57 -0700

Hi, Fred,

On 7/1/2013 1:27 PM, Templin, Fred L wrote:
...
> IPv6 transition mechanism "best practice".

I was giving an example of IPv4-in-IPv4, to demonstrate that when minMTU 
== minMRU then tunnels are not possible.

Agreed.

> one example on the IPv6 list).

We agree, but the punchline is that:

a) minMRU *MUST* be larger than minMTU, or else there cannot be tunnels

	how much larger depends on how much nesting you expect;
	IPv6 allows 5 levels if plain nesting and around 2 of IPsec.
	The former seems fine, but the latter is tight. I would not
	at all be surprised if future networking ended up experiencing
	more than 2 levels of IPsec tunnel.

b) we CANNOT deprecate fragmentation

	if we do, we are deprecating tunneling

> (i.e., a routing loop). SEAL allows up to 8 levels of nesting.

It's "recursive" when it's recursive, and there's strictly no way to 
determine that because an address can have different meanings at 
different layers. The only solution would be to have each layer insert a 
"thumbprint" and look for cycles in that, but that approach would 
consume the encapsulation budget more quickly.

Two levels of tunnel are required to allow revisitation, where a single 
machine can emulate multiple virtual routers (without requiring OS 
virtualization such as VMware). More levels are needed to support other 
kinds of abstraction; my own work used up to 4 levels for 
non-demonstration purposes (we demo'd 16).

> assume 1500 when the specs say they can only assume 576.

Agreed. That's why we need *realistic* numbers for that, not merely 
those that reflect what's implemented (e.g., for IPv4).

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 21:18:04 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 1:51 PM
> To: Templin, Fred L

I took a quick look, and it looks like RFC2003 skirted the issue
of minMTU/minMRU sizes altogether. At least the v6/v4 transition
mechs did a cursory study of packet sizes.

> > So, we can't assume an unlimited reassembly buffer on the

OK.

> a) minMRU *MUST* be larger than minMTU, or else there cannot be tunnels

Yes - I am totally with you on this.

> 	how much larger depends on how much nesting you expect;

Not necessarily. If the first tunnel ingress set a small fragment
size (say, 1280) and fragments anything between (1280-1500) to
something smaller (say 750), then each subsequent ingress in the
path would not need to fragment further and we can get away with
a reassembly buffer size of just (1500+HLEN) at the ultimate tunnel
far end.
 
> 	more than 2 levels of IPsec tunnel.

Right, but without fragmentation the way you *stack* those levels
of nesting needs to be carefully coordinated which may not always
be possible.

> 	if we do, we are deprecating tunneling

If we deprecate IP fragmentation but then at the same time replace
it with SEAL we will still have the functionality that we need.
 
> consume the encapsulation budget more quickly.

SEAL has a "level" field which is essentially a counter that counts
down from 7 to zero. When we reach level 0, assume recursion and quit.

> non-demonstration purposes (we demo'd 16).

Sounds very reasonable. I think we want to avoid getting locked
into rigid notions of what tunnel nesting would be used for.

> those that reflect what's implemented (e.g., for IPv4).

Right, but what I am ultimately after is an unlimited tunnel
MTU (I think you were on board with this idea from earlier
discussion). The only difference is that I want fragmentation
and reassembly out to 1500 bytes only, and then let the bigger
packets go through unfragmented as long as they fit. "Take care
of the smalls, and let the bigs take care of themselves."

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 14:34:23 -0700

Cutting to the chase...

> it with SEAL we will still have the functionality that we need.

That's replacing fragmentation with fragmentation. The only reason to 
want to get rid of fragmentation is to simplify endpoints and/or 
simplify forwarding that does DPI. SEAL won't help either of those.

Further, we *have* fragmentation now. Getting rid of it would be 
premature until SEAL is required and widely deployed.

>> >Agreed. That's why we need*realistic*  numbers for that, not merely

We don't disagree on IPv6. My claim is that for IPv4 it is *necessary* 
that endpoints support MTUs lower than 567 to allow IPv4 fragmentation 
to support IPv4-in-IPv4 tunnels, including IPsec.

There are two solutions, of course - deploying SEAL would be the other. 
But I would expect that fixing incorrect implementations of an existing 
standard would be a lot easier than getting new code deployed for 
something that isn't yet standard.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 21:50:07 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 2:34 PM
> To: Templin, Fred L

SEAL helps with the DPI because it makes sure that the ULP headers
appear in the first fragment - there is no possible way to construct
a "runt" first fragment with SEAL. SEAL also takes care of the
overlapping fragments attack vector. There is a growing laundry list
of other complaints about IP fragmentation (e.g., Identification field
length and ID setting strategy) that do not apply to SEAL. 
 
> premature until SEAL is required and widely deployed.

SEAL is compatible with IP fragmentation, so we can transition
to SEAL without having to first decommission IP fragmentation.
So, maybe the document should advocate "SEAL transition" rather
than "IP fragmentation deprecation". 

> to support IPv4-in-IPv4 tunnels, including IPsec.

I think there is way too much deployed base out there now
that boldly assumes 1500 that it would be impossible to put
them back in the bottle now.

> something that isn't yet standard.

There are too many compelling reasons to migrate away from IP
fragmentation and move toward SEAL. Maybe also look at some of
the discussion on the IPv6 list so we can avoid duplication.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 15:21:08 -0700

On 7/1/2013 2:50 PM, Templin, Fred L wrote:
> appear in the first fragment

You can't know that unless you parse the IPv6 header chain. And adding 
SEAL inbetween IPv6 and the inner IPv6 adds one more step to that chain.

Either way, DPI has to follow the chain step-by-step, vs. in a single 
leap as with IPv4.

> - there is no possible way to construct

Yes, but support for (vs not interfering with) DPI isn't one of them.

> than "IP fragmentation deprecation".

You can talk to Ron about that. But AFAICT that won't solve the overall 
problems his draft is seeking to address (whether they can or should be 
addressed is debatable, IMO).

...
...
> them back in the bottle now.

Then your table should have said IPv4 has a minMRU of 1500, but it 
doesn't. That's widely deployed, but cannot be assumed. Esp. on 
low-powered/low-mem devices.

> the discussion on the IPv6 list so we can avoid duplication.

I agree that SEAL is better than IP at fragmentation, but it's still a 
network fragmentation layer when used for IP tunnels. It doesn't solve 
the issues that some are concerned about recently.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 22:51:28 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 3:21 PM
> To: Templin, Fred L

The SEAL first fragment MUST contain at least 256 bytes (or up to
the end of the packet) according to the specs. If that is not large
enough to include sufficient ULP headers then the packet should
probably be dropped anyway.  

> Either way, DPI has to follow the chain step-by-step, vs. in a single

Right, but how long can those chains be and still be considered
a "realistic" IPv6 packet?

> Yes, but support for (vs not interfering with) DPI isn't one of them.

This comes to whether it is possible to set a maximum upper
bound on the length of a chain of IPv6 extension headers. And,
if 256 is too small SEAL the spec can still be changed to push
it to 512.

Maybe what is needed is a specification of the maximum IPv6
header length the same as we have a specification for the
maximum IPv4 header length? I would propose 256 bytes, but
I could live with something somewhat larger like 512.

> low-powered/low-mem devices.

About the 1500/576, I'm just repeating what I read in RFCs 791
and 1122. I think we need to stop debating about IPv4 tunnels.
They're widely deployed, and we can't put them back in the bottle.

> the issues that some are concerned about recently.

As we have discussed in the ipv6 list, I am proposing SEAL as a
general fragmentation/reassembly framework for all applications
and not just tunneling. The SEAL header has the same footprint
as IPv6 frag header and can be used for the same purpose, whether
it be in tunnel-mode or transport-mode.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 16:20:01 -0700

Fred,

> a "realistic" IPv6 packet?

Although I agree it's important to have the header you want to examine 
in the first fragment, you still need to parse the chain.

With IPv4 that's one opcode - jump based on the offset in a memory.

With IPv6, you need to run that opcode as many times as there are 
headers. That's the problem. Not whether you have the header you want to 
examine, but whether you can access it in a single hardware operation.

That's a problem with or without SEAL, and it's the problem that the 
current discussion on intarea is trying to address.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 1 Jul 2013 23:30:53 +0000

Hi Joe,

> Sent: Monday, July 01, 2013 4:20 PM
> To: Templin, Fred L
> in the first fragment,

Good. That means we need to set an upper bound on the maximum
length of an IPv6 header chain, right? And remember, with SEAL
the minimum first fragment size is 256 bytes whereas the min
first-frag size for IPv6 fragment header is only 8 bytes. SEAL
therefore naturally avoids an attack vector available for IPv6
frag/reass.

> you still need to parse the chain.

That is by design of the IPv6 protocol, and not something that
can be changed now regardless of the encapsulation, right?

> With IPv4 that's one opcode - jump based on the offset in a memory.

Right.

> current discussion on intarea is trying to address.

Have there been any proposals? As in, is there any way to approach
this without a major redesign of IPv6? I also thought that the IPv6
"quadword-aligned" requirement addressed performance. Is that not
helpful?

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Mon, 01 Jul 2013 16:53:55 -0700

Hi, Fred,

On 7/1/2013 4:30 PM, Templin, Fred L wrote:
> Hi Joe,
>
>...
>>>>> Either way, DPI has to follow the chain step-by-step, vs. in a
>> single leap as with IPv4.
>>>
> length of an IPv6 header chain, right?

That is only part of the solution. The other part is a way to jump to 
the place you want to quickly, but that makes it hard to modify the chain.

> That is by design of the IPv6 protocol,

Yes.

> and not something that

Well, some are trying to "change" it by deprecating IPv6 options 
completely - including fragmentation. That would mean that 
non-encapsulated packets would have a zero-length chain.

It doesn't help packets with encapsulation - for SEAL, IP-in-IP, etc.

> "quadword-aligned" requirement addressed performance. Is that not

The primary proposal is to deprecate all IP options.

Alignment addresses one part of the problem - how fine-grained the jumps 
are - but not the chain.

--

My point is that SEAL is not a solution to the problem driving the 
desire to get rid of IP fragmentation. That desire isn't because 
fragmentation is incomplete or suboptimal, it's because it's *there*.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 2 Jul 2013 15:06:03 +0000

Hi Joe,

It occurred to me that the SEAL header has an available byte
that could be used as an offset to branch to the end of the
extension header chain without having to parse all other
extension headers in between. If that's not good enough
then I don't know what else to say, because we have already
established that some form of fragmentation is necessary.

Please don't answer if it is uncomfortable to do so, Joe,
but are you against seeing IPv6 move forward?

Thanks - Fred

> Sent: Monday, July 01, 2013 4:54 PM
> To: Templin, Fred L
> >> headers. That's the problem. Not whether you have the header you
> want
> >> to

---

From: Joe Touch <[email-masked]>
Date: Tue, 02 Jul 2013 14:19:41 -0700

Hi, Fred,

> but are you against seeing IPv6 move forward?

I'm not, but this won't help.

I had thought about this as a possibility for a new IP option, i.e., one 
that jumps to the end of the options. However, that option would need to 
be updated by any code that added an intermediate header, and I'm not 
sure how that would work.

IP already puts all the options that a router should be looking at in 
one place - by design. The problem here is that routers want to look 
further. In IPv4, there's a notion of the base header, option headers, 
and a payload. In the 'old days', the payload was the transport header, 
so it was easy to know where to look to find port numbers.

Those days have changed. There are many protocols that add 'shim' layers 
between IP and transport, or other places in the conventional stack. 
That's part of why IPv6 designed its headers as chained - to make it 
easier to support shim layers without redesigning IP and without 
limiting the space available for such layers.

So we have a tension between the many folks who want to insert these 
shims and DPI devices. The DPI devices want to do less work than an 
endpoint to find the transport information, but that may or may not be 
feasible using the current IPv6 structure.

Joe