Subject: [RRG] MTU/fragmentation AGAIN

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 23 Dec 2007 00:51:54 +0100

Sorry about that.

As I said a few days ago, Fred's fragmentation model makes a lot of  
sense if we want to allow fragmentation in the first place. Let's see  
where we stand if we don't want to allow fragmentation.

If the path between the ITR and ETR supports 1500 bytes +  
encapsulation there shouldn't be any issues with path MTU discovery  
that aren't there without the encapsulation, too. (I'm using LISP  
terminology but this applies to all map/encap schemes.)

There are two possible other cases:

- the path doesn't support 1500+E and the ITR knows this
- the path doesn't support 1500+E but the ITR doesn't know this

In the first case, the ITR can simply return a too big message, and  
hosts/sites implementing PMTUD correctly won't have a problem. Hosts/ 
sites that don't will experience a PMTUD black hole. I'll get back to  
this.

In the latter case, the ITR would have to do PMTUD towards the ETR and  
after that send too bigs based on that result to the source hosts that  
send packets through the ITR. This is unwanted complexity for the ITR  
and it means that a relatively high number of packets that flow  
through the ITR could be dropped during this process. As such, I'd say  
that it's probably unacceptable to design a network such that this  
situation is common. I.e., having an ITR on network A that supports  
1500+E and an ETR on network B that supports 1500+E but then have the  
interconnection between these networks happen over an internet  
exchange with a 1500-byte MTU would trigger PMTUD and lost packets on  
ALL sessions between these ISPs, which I wouldn't find acceptable. In  
other words: internet exchanges used between two 1500+E networks that  
do map/encap must be upgraded to support at least 1500+E.

The reason why PMTUD works so badly today is most likely because it  
doesn't have to: all hosts connected to the internet that advertise  
TCP MSS of 1500 - overhead can successfully receive 1500-byte IPv4  
packets with DF=1. However, if a site wants to deploy an ITR and/or  
ETR, and suddenly, PMTUD black holes happen, the site has a very  
strong incentive to make the changes necessary to make those black  
holes go away. In the case of an ITR, this means making sure the  
source host receives the too big messages. In the case of an ETR, this  
means announcing a smaller TCP MSS.

So I'd say that SITES should be able to deploy xTRs even though they  
can't support a 1500+E MTU. (There needs to be a way to communicate  
the ETR MTU limiation back to ITRs, though.) However, the same is not  
true for ISPs: in that case, the ISP operators create the problem, but  
site operators / host admins (for a large number of sites) need to fix  
the problem. This is something that isn't going to work in practice.  
So ISPs MUST support 1500+E both on their xTRs and on peering links to  
other ISPs that run xTRs.

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Brian Dickson <[email-masked]>
Date: Wed, 02 Jan 2008 15:19:32 -0500

> where we stand if we don't want to allow fragmentation.
Here, here! :-)
> - the path doesn't support 1500+E but the ITR doesn't know this
Just to dig a bit deeper, is my understanding on PMTUD correct, as follows?

The MTU to be discovered from A->B is not necessarily related to the MTU 
to be discovered B->A
A needs to know MTU-A for A->B, and vice versa, to avoid fragmentation.
If A can do PMTUD, A does (or should) not depend on B in any way to 
accomplish this.

So, PMTUD can be considered two simplex problems - A->B and B->A.

Okay so far?

But, with encaps/decaps, we have A->ITR->path-1->ETR->B,
which can change to A->ITR->path-2->ETR->B, at any time.
And it is possible that MTU(path-1) != MTU(path-2).

Does this do bad things to PMTUD?

If we are able to produce a newer, better, PMTUD, that requires 
deployment of new TCP stacks, and achieves 99.9%  of the desired benefit 
(no fragmentation), when deployed only on the server-side of things, 
would that be worthy of spending time on?

Here's what I'm thinking:

For TCP, all the window-size stuff is based on bandwidth*delay. Number 
of packets and rate of transmission.

If the slow-start and back-off stuff, also used MTU as a parameter, 
PMTUD could become an implicit part of TCP.

By shrinking packet size, the effective window shrinks. Grow packet 
size, grow window. Adapt the packet size first, then the rate, to 
achieve the optimum TCP window.

And, if MTU is exceeded, without having (or needing!) any ICMP or 
fragmentation, packet loss would result in lowering of MTU until the 
right MTU was used.

It's a bit wasteful in the presence of true PMTUD, but in its absence, 
it makes things "just work".

Thoughts?

Brian Dickson

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 2 Jan 2008 13:51:53 -0800

Brian,

Thanks for the thoughts, but you are not saying anything
here that isn't already well understood and/or has been
said many times in the past. First, path MTU from A->B
does not necessarily have any relation to path MTU from
B->A; it is well understood that the paths can easily
be asymmetric and so the MTU discovery is simplex in
both the A->B and B->A directions. As a result, the
paths have to be probed with a large probe request sent
in the forward direction and a tiny probe reply returned
in the reverse. Sprite-mtu defines a way of doing just
that - but this gives me an idea for a minor change which
entails respecifying the sprite-reply to be minimum-sized
rather than requiring perishable padding for the
sprite-request. Thanks for that.

About MTU changes due to path changes, that is a problem
that needs to be dealt with in any case and not just the
map-and-encaps case. The two ways of detecting this are
to begin receiving packet-too-bigs when none were
received previously, or to proactively probe the path,
or both. (In the map-and-encaps case, sprite-mtu probing
can be used for proactive probing by the ITR.)

About end-to-end MTU involvement from TCPs, RFC4821 already
provides a published standards-track specification for just
such a method.

Thanks - Fred
[email-masked] 

> -----Original Message-----
> From: Brian Dickson [mailto:[email-masked]] 
> Sent: Wednesday, January 02, 2008 12:20 PM
> To: Iljitsch van Beijnum
> Cc: Routing Research Group list
> Subject: Re: [RRG] MTU/fragmentation AGAIN
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 2 Jan 2008 14:40:57 -0800

> can be used for proactive probing by the ITR.)

FWIW, some might suggest that another way to deal with
this is to set DF=0 in all packets after the path has
been probed and just let it fragment if an MTU restriction
due to a path change occurs. But, RFC4963 presents some
sobering insights as to why that would be a bad idea.

Thanks - Fred

---

From: Brian Dickson <[email-masked]>
Date: Wed, 02 Jan 2008 19:45:52 -0500

Templin, Fred L wrote:
> both the A->B and B->A directions.
Yep, I'm well aware. I worked with Geoff Huston on some asymmetric BGP 
stuff,
in the late 1990's, involving simplex satellite links from Canada to 
Australia,
between Teleglobe and Telstra. Now *that* was asymmetric. :-)
> but this gives me an idea for a minor change which
Glad to help. :-)
> can be used for proactive probing by the ITR.)
The problem I see there, specific to map-and-encaps, is the detection of 
path MTU changes.
The xTRs are the only things to get the PTB's, and the hosts are the 
ones who need to do PMTUD.

Probing can detect the new MTU, but what triggers the probing?
> [email-masked] 
Thanks. I wasn't aware of 4821.

There was some interesting tidbits in there, that piqued my interest.

One of those was "IP Host Fragmenting" - avoiding middle-of-the-path 
fragmentation.

It could be very useful as an implementation scheme when probe packets 
are lost
due to MTU exceeded, where no PTB ICMP is received.

Specifically, sending the probe packet again as two fragments, with DF=1 
set.
It guarantees no further fragmentation, validates the PMTU, and avoids 
having to
do anything other than a retransmit of an existing packet.

Do you know if anyone is working on implementations of 4821? I'd like to 
pass this
idea along to them...

Brian
>>> sense if we want to allow fragmentation in the first place. 
>>>       
>> Let's see 
>>

---

From: Brian Dickson <[email-masked]>
Date: Wed, 02 Jan 2008 20:30:18 -0500

> sobering insights as to why that would be a bad idea.
I've been thinking about a variation of the map-and-encaps style of LISP.

(Obviously the control plane and data plane are orthogonal, and don't 
share much
other than the need for the control plane and EID->RLOC mapping.)

I was thinking about this at the last IETF, and think I have a better 
handle on LISP now.
(Thanks, Dino.)

(The following is, finally, a way to put the "pure math" part of my 
degree to work. :-))

There's two kinds of "mapping" in math that are interesting: "1:1", and 
"onto".
What we conventionally think of as "1:1", is, technically, "1:1 *and* onto".

I bring this up because of an interesting observation. Regardless of 
RLOC used,
if mappings are looked up by xTR's in both directions, RLOC->EID at the ETR
is completely deterministic.

So, let's consider an interesting scenario, most feasible in IPv6 
because of address scarcity in IPv4.

What if the EIDs were (assigned | issued) as PI address blocks? 
Specifically, PI blocks not routed into the DFZ,
per se (but potentially available as an "upgrade path"), as an 
alternative to ULA or ULA-{C|G}.

The concept is to "map-and-map", from PI-EID to PA-RLOCs, via 1:1 
mapping(s).
The reverse mappings are likewise 1:1,  reasonably trivial, and 
essentially unchanging regardless of RLOCs in use.

The real benefit is that, like SIIT, it is feasible to do ICMP 
transliteration, thus support PMTUD and similar activities.
That, and not affecting the PMTU itself in the first place. :-)

The map-and-map effectively becomes NAT + SNAT, without any port mangling.

The only substantial difference I can think of, is the RLOC reachability 
bit vector.

The main question I have is, is there any obvious way to maintain the 
benefit of that bit vector without having to go all the way to 
encapsulation?

Brian Dickson

P.S. The mechanisms might even support non-LISP-to-LISP connectivity, 
presuming LISP-aware host stacks.)

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 3 Jan 2008 08:28:15 -0800

Brian,

> Sent: Wednesday, January 02, 2008 5:30 PM
> To: Templin, Fred L
> Cc: Iljitsch van Beijnum; Routing Research Group list; Dino 
> Farinacci; Tony Li
...
> That, and not affecting the PMTU itself in the first place. :-)

Among the myriad issues that have us scratching our heads,
SIIT doesn't work when a middlebox returns an ICMP PTB that
only includes the minimum 8 bytes beyond the IP header of
the packet-in-error. FWIW, net/ipv6/sit.c in the Linux
kernel distro has the following to say about SIIT:

+ #ifndef I_WISH_WORLD_WERE_PERFECT
+ 
+ /* It is not :-( All the routers (except for Linux) return only
+    8 bytes of packet payload. It means, that precise relaying of
+    ICMP in the real Internet is absolutely infeasible.
+ */

Fred

---

From: Scott Brim <[email-masked]>
Date: Thu, 3 Jan 2008 14:16:21 -0500

Excerpts from Brian Dickson on Wed, Jan 02, 2008 08:30:18PM -0500:
> There's two kinds of "mapping" in math that are interesting: "1:1",
> and "onto".  What we conventionally think of as "1:1", is,
> technically, "1:1 *and* onto".
>
> RLOC used, if mappings are looked up by xTR's in both directions,
> RLOC->EID at the ETR is completely deterministic.

You lose me here already.  At the ETR there is no RLOC->EID *mapping*.
If the RLOC is a valid address of the ETR, the packet is decapsulated
and forwarded toward the EID in the inner header.  EID->RLOC is
determined at the ITR, but I don't understand when you say that
RLOC->EID is determined at the ETR.  RLOCs name xTRs.

But anyway ...

> Specifically, PI blocks not routed into the DFZ, per se (but
> potentially available as an "upgrade path"), as an alternative to
> ULA or ULA-{C|G}.
>
> mapping(s).  The reverse mappings are likewise 1:1,  reasonably
> trivial, and essentially unchanging regardless of RLOCs in use.
>
> transliteration, thus support PMTUD and similar activities.  That,
> and not affecting the PMTU itself in the first place. :-)
>

So ... setting terminology aside, you're suggesting a 1:1 relationship
between an aggregatable, globally routed address and a locally scoped,
not necessarily aggregatable address?  

If I understand you correctly, why have that kind of mapping at all?
Why not do something like 8+

---

From: Brian Dickson <[email-masked]>
Date: Thu, 03 Jan 2008 15:16:40 -0500

> not necessarily aggregatable address?  
The idea for this is that the EID, while not announced globally, is 
still globally scoped.

This is so that it is possible for end-sites to move in, out, or 
between, EID space and DFZ space,
as they grow, shrink, or move.

And on the RLOC:EID side, it's more of a N*(1:1) relationship, where N 
varies by EID block,
as well as over time, including which specific RLOC blocks are used per EID.
> Why not do something like 8+
It's a question of flexibility and scope.
It's almost exactly analogous to the major problems with TLA/NLA from an 
operator perspective.

There is no need to impose additional structure, beyond the existence of 
prerequisites for use of the technology.
Imposing any fixed structure, universally, is very likely to be 
universally opposed (and rightly so) by v6ops folks.

The idea is that RLOC space is arbitrarily subnetted by LIRs, in 
whatever VLSM scheme they need, and that
the size and number of end-assignments is also arbitrary.

EID end-sites would clearly best benefit from having all RLOC blocks be 
the same size, but as long as the EID
end-site is able to operate from within some smallest-common-denominator 
block, all is good.

The main benefit I see, is that the EID site is able to entirely control 
the admistration and operation of the ITR/ETR,
if they want, without needing any or all of their upstream providers to 
do anything more than assign PA blocks to them.

I'm sure both the ISP and the EID site would be very happy to have that 
as one possible arrangement, even if it is also
possible for the EID site to out-source operation of the xTRs to any 
third party, including the ISP.

Brian

---

From: Brian Dickson <[email-masked]>
Date: Thu, 03 Jan 2008 19:54:16 -0500

Templin, Fred L wrote:
> the packet-in-error.
Hmm. I can see how that PTB issue would break SIIT.

However, when doing 1:1 NAT (no port changes, only swapping prefix 
portions of src and/or dst),
it does not appear (to me at least) to cause any problems.

The PTB message would get sent to the NAT'd sender address, which the 
ITR would need to properly
un-NAT (including the payload IP header src/dest).

That un-NAT function, however, would be completely deterministic, under 
the proposed scheme.
RLOC->EID would be guaranteed to be a 1:1. (Several RLOCs can map to the 
same EID, of course...)

At least, I think this is the case. If the real sender, sending with its 
own EID as src, receives a PTB, and the IP header
inside the PTB matched the header of a sent packet, that should be 
enough for PMTUD to work, should it not?

And since the packet sizes wouldn't change, the MTU of the PTB would 
still be valid.

Am I missing something?

(This is IPv6-to-IPv6 mapping, effectively 1:1 NAT on src/dest, rather 
than tunneling packets. No S*IT is involved.)

Brian

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 3 Jan 2008 17:06:24 -0800

Brian,

I probably didn't look at your proposal carefully enough,
and I may not have time to soon. But, regardless of
whether/not PTBs could be translated there is still a
trust issue. We can probably (hopefully!) trust PTBs from
a router within the same site (i.e., intra-domain). But,
the same may not be true for PTBs that come from outside
of the site (inter-domain) and these may in fact be used
as a DOS vector. To my understanding, inter-domain is the
deployment case for the map-and-encaps architectures we
have been discussing?

Thanks - Fred

> Sent: Thursday, January 03, 2008 4:54 PM
> To: Templin, Fred L

---

From: Brian Dickson <[email-masked]>
Date: Thu, 03 Jan 2008 20:42:56 -0500

Templin, Fred L wrote:
> as a DOS vector.
Yep, same as ICMP in IPv4 is a (potential) DOS vector.

However, in both cases, I believe that the fact that the original header 
is included,
makes it feasible for the host receiving the ICMP packet to (a) only 
handle them
when it is doing PMTUD, and (b) validate them against transmitted packets.

Or, stateful packet inspection (firewall) devices could be taught the 
"rules" for matching
PTB against packets that were sent (but are TB).
> To my understanding, inter-domain is the
For some definition of inter-domain, yes.

For instance, two customers of ISP A, who are both multihomed to other ISPs,
and exchanging LISP traffic. No BGP, so not "inter domain routing" per se,
but inter-"domain" in the sense of crossing third-party infrastructure.

The benefit of LISP is to the multi-homed entity, and only seen when 
other parties
use LISP to communicate with them. In that sense, the entity *wants* 
everyone
to know about the EID->RLOC mappings, and to allow use of that pathway,
modulo any security policies (firewall etc.)

Brian