Subject: [RRG] Sprite & IPTM while PMTU probing is in progress

From: Robin Whittle <[email-masked]>
Date: Wed, 28 Nov 2007 13:09:04 +1100

Hi Fred,

Here is a diagrammatic explanation of how IPTM would works, and a
question about how Sprite would work.  I haven't been able to
understand your ID clearly enough to answer this myself.

  http://tools.ietf.org/html/draft-templin-inetmtu-06
  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

IPTM only sends a PTB message to the SH after it has reliably
ascertained the PMTU to the ETR - and then only when the SH sends it
a packet which is of a length which would exceed that PMTU, once it
was encapsulated.  Until then (with the first such "long" packet,
and with any others which arrive while the ITR is probing) "long"
outer packets are fragmented, whether or not the inner packet has
its DF flag set, to be reassembled at the ETR before decapsulation.

A "long" inner packet is one which, once encapsulated, would exceed
the ITR's current best estimate of the PMTU.  This would initially
be a default such as 1280 bytes.  If this default value is above the
minimum for the protocol, eg. 576 for IPv4, then this value of PMTU
to the "core" of the net must be available to every ITR and ETR and
would be part of the specification of the ITR-ETR scheme.

  (Did you mean "1280" instead of "1500" in:
   "For IPv6, the minimum EMTU_R is 1500 bytes ..."
   http://psg.com/lists/rrg/2007/msg00623.html ?)

The default value would be replaced by a higher value once the probe
process was complete.  The pattern would be something like this,
assuming the SH's initial idea of PMTU to the Destination Host was
1500.  I will assume an ENCAPS overhead of 20 bytes (as with IPv4
Ivip, though other ITR-ETR schemes have higher overheads) and that
all ITRs and ETRs are located so they have an MTU of at least 1280
from the DFZ.

   Inner      ITR action on           SH's idea  DH gets
   packet     outer packet            of PMTU
   length     following encapsul-     to DH
              ation of inner packet

                                      1500

   200       Send outer packet -      1500       The packet
             the length is less than
             1280.

  1500       Fragment packet and      1500       The packet
             commence probing                    (Less efficient
                                                 and more error-
                                                 prone tunnel with
                                                 2 packets instead
                                                 of 1, but this is
                                                 only for a few
                                                 seconds, I hope.)

  1500       Fragment packet and      1500       The packet
             continue probing

             ... etc.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

  1400       Send outer packet -      1500       The packet
             the length is <= 1460.
             (This length would not
             necessarily be sent - it
             is just to show that the
             ITR will now send longer
             packets without frag-
             mentation than before.)

  1500       Drop the packet and send
             the SH a PTB message
             with value 1440.         1440       Nothing, but the
                                                 ITR is usually
                                                 close to the SH,
                                                 and it doesn't
                                                 take long for...

  1440      Send the outer packet -  1440        The packet
            the length is <= 1460                (Now the tunnel
                                                 is handling
                                                 optimal length
                                                 packets.)

This pattern would continue unless the ITR, with periodic probing,
decides that the PMTU is less than 1460 (it might do this quickly if
it got a PTB message from a router in a new, more MTU-challenged,
path to the ETR), and if the SH sends a packet which would be too
big for the new lower value of PMTU.  Then the ITR would send
another another PTB message to the SH, with a lower value than 1440.

Alternatively, occasional probing by the ITR might discover a higher
value of PMTU to this ETR, and the SH could discover this increase
by trying its luck with a larger packet - and either having it
accepted, or rejected with a PTB containing the new higher value,
minus 20.

Sprite, as I understand it for IPv4 ...

  (Fred, the "Section 5.5.x" numbers in Figure 2 should be "5.6.x"
   except for 5.5.6, which should be 5.6.5.  Other references
   in the ID to "5.5.4" etc. may also need correction.)

... actually, I don't understand it clearly enough to describe it.

I think Sprite will fragment outer packets, as does IPTM,
irrespective of the DF flag of the inner packet.  The criteria for
which IPv4 outer packets are fragmentable is complex (5.6.4).

I am not sure how Sprite handles "large" outer packets while it is
probing.  Does it fragment them as IPTM does?  Or does it do the
following, which is the same as what it does for a "long" packet
once the PMTU has been reliably ascertained:

  (5.6.5) "... admits the packet but also sends a PTB message ..."

It seems strange to me to send the packet (unfragmented, I assume)
while also sending back a PTB message to the sending host.  Wouldn't
this cause needless traffic and/or confusing signals to the SH if
the outer packet does in fact arrive at the ETR and therefore the
inner packet is delivered to the destination host?

Here I will assume IPv4 only, with 1280 bytes for the default PMTU
for every ETR the ITR has not yet probed.  I will also assume an
encapsulation overhead of 20, although this would typically be
higher for Sprite and non-Ivip ITR-ETR schemes.

If the ITR sends a PTB message to the SH when the first packet (or
multiple packets) length exceeds the default PMTU value and then,
after probing, decides the PMTU is 1480, then I am concerned that
the SH would get contradictory values in these PTB messages.

At first the SH would be told to send packets no longer than (1280 -
20 = 1260) and later, it would be told to send packets no longer
than (1480 - 20 = 1460).

But if the SH took notice of the first PTB message, it probably
wouldn't send any longer packets which would trigger the second.
So, if my understanding of Sprite is correct, the SH would
experience something like this:

   Inner      ITR action on           SH's idea  DH gets
   packet     outer packet            of PMTU
   length     following encapsul-     to DH
              ation of inner packet

                                      1500

   200       Send outer packet -      1500       The packet
             the length is less than
             1280.

  1500       Send packet and                     Probably nothing
             commence probing                    - however, if the
             Send PTB with value                 packet was a little
             1260                     1260       shorter, such as
                                                 1440, then it may
                                                 arrive at the ETR
                                                 in one piece,
                                                 despite the SH
                                                 being told it was
                                                 too big.

             SH breaks the message
             into smaller packets
             and retries:

  1260       Send packet and          1260       The packet
             continue probing

             ... etc.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

  1260       Send outer packet -      1260       The packet
             the length is <= 1460.

  1260       SH would probably keep              The packet -
             sending packets of                  but more and
             length <= 1260.  Unless             shorter
              the SH was pushy, it               packets than
             would never discover the            the ITR-ETR
             PMTU it could use was               tunnel can
             in fact 1440.                       handle.

  - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 28 Nov 2007 11:41:45 -0800

Robin,

Please note that the name of the proposal is "sprite-mtu";
not "sprite". See below for some responses:

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]] 
> Sent: Tuesday, November 27, 2007 6:09 PM
> To: Routing Research Group list
> Cc: Templin, Fred L
> Subject: Sprite & IPTM while PMTU probing is in progress
> 
> Hi Fred,
> 

On- and off-list discussions have explored the idea of
requiring a 2KB EMTU_R on all ETRs and accommodating all
1500-byte and smaller packets; even if fragmentation is
needed and the inner DF=1. This would uphold the "principle
of least surprise" to the SH, but the issue comes in
knowing the EMTU_R of the ETR and in avoiding excessive
fragmentation. 

Since the ITR cannot know the EMTU_R of the ETR a priori
unless there is some spec that says: "all ETRs MUST configure
an EMTU_R of at least X bytes", the ITR should not simply
fragment the outer packets (or, allow the network to fragment
them) since they could black-hole. Also, a burst of packets
requiring fragmentation that arrive before the path is probed
can cause fragment misassociations at the ETR which could
result in undetected corruption per RFC4963.

There are also other factors to consider, including that
the ITR may not have ultimate control over the setting of
the ip_id. And, the ETR may not be able to receive non-initial
fragments in the first place. (These factors can be mitigated
by the placement of the ITR and ETR in some use cases, however.)

> be a default such as 1280 bytes.
>
> If this default value is above the

The pathMTU cannot be known a priori; all that can be known
a priori is the EMTU_R of the ETR if there is a specification
for the minimum size.

>    http://psg.com/lists/rrg/2007/msg00623.html ?)

1500 per RFC2560, Section 5.
 
> all ITRs and ETRs are located so they have an MTU of at least 1280

Do you mean to say "pathMTU" or "linkMTU"? In terms of
"pathMTU", do we need to consider links with configurable
linkMTUs that might have either mis-configured or overly-
conservative values? 
 
>   1500       Fragment packet and      1500       The packet

This could black hole and appear as congestion-related
loss to the SH.
 
>              with value 1440.         1440       Nothing, but the

Depending on the placement of the ITR, this PTB might not
make it back to the SH.
 
> path to the ETR), and if the SH sends a packet which would be too

There is not strictly any periodic probing needed to detect
pathMTU reductions, since the data packets serve as virtual
probes. The data packets will be lost and might be considered
by the SH as congestion-related loss if the PTB can't be
translated by the ITR and sent back to the SH. But, the ITR
will be able to return the correct PTB when the SH retransmits.
This is the same as for sprite-mtu. 
 
> accepted, or rejected with a PTB containing the new higher value,

SHs that don't implement RFC4821 will have to wait for
a long time before trying a larger packet (RFCs 1191 and
1981 say 10min, I believe). SHs that implement RFC4821
can retry end-to-end probing more frequently than that,
since loss of a probe does not expose data to silent loss.  

>    in the ID to "5.5.4" etc. may also need correction.)

I'll fix these editorials.
 
> irrespective of the DF flag of the inner packet.

That is true.

> The criteria for

That text borrows from the tunnel MTU and fragmentation
discipline set forth in RFC4213, Section 3.2. But, I don't
know what you mean by complex? 
 
>   (5.6.5) "... admits the packet but also sends a PTB message ..."

It does the latter; this borrows from RFC2003, Section 5. 
 
> inner packet is delivered to the destination host?

To the SH, it would appear that there is a router on the
path returning inaccurate information. This can happen
already today, since routers can be misconfigured, and
spoofed PTBs can be sent from any node in the network.

SHs that implement RFC4821 should not have a problem
deconflicting the (suspect) PTB information from (authentic)
end-to-end feedback from the DH, but should benefit from the
PTB info when the actual data is not delivered to the DH.

ITRs can help the situation by sending sprites of, e.g.,
1500 bytes into the tunnel early in the process so that
most if not all SHs that use the tunnel will see a 1500
byte or larger MTU. 
 
> higher for Sprite and non-Ivip ITR-ETR schemes.

I don't understand "higher for sprite-mtu"? 
 
> 20 = 1260) and later, it would be told to send packets no longer

Note: in the next draft version I would like to rewrite
the second bullet of Section 5.6.4 as:

   o  for IPv4/*/IPv4 tunnels, 'pathMTU' is less than
      MIN(EMTU_R, 1280+ENCAPS) bytes and the inner IPv4
      packet is no larger than MIN(EMTU_R-ENCAPS, 1280).
 
>    200       Send outer packet -      1500       The packet

Probing can start here also.
 
>              1260                     1260       shorter, such as

With the note above, the returned size would be 1280;
not 1260. Also, I'm not sure as to the "probably nothing"
as linkMTUs increase above 1500 (perhaps someone could
send the IEEE reference that proposes the increase for
802 linkMTUs).

>   1260       Send packet and          1260       The packet

By probing, do you mean by the ITR or by the SH? I am
assuming that SHs will begin using RFC4821 and will probe
the path for themselves independent of any probing done
by the ITR.

>              PMTU it could use was               tunnel can

IMHO, SHs that use RFC4821 can be "pushy" within reason.

I would like to add one other note about the 1280. That
number comes from the SHOULD in RFC4213, Section 3.2.1.
The reason I am taking the SHOULD is that there can be
additional encapsulations on the path between the ITR
and ETR (e.g., tunnel-mode IPsec) and we don't want to
cause fragmentation for those, either. If the ITR and
ETR are arranged such that there will be no additional
encapsulations on the path (and the ITR has a way of
knowing this) then the spec could use the RFC4213,
Section 3.2.1 "configuration knob" to push the 1280 up
to as much as 1480 or perhaps even 1500. I would not
want to go any higher than this, since it could involve
excessive fragmentation resulting in undetected data
corruption. Maybe I should add something about this
to the spec?

Fred
[email-masked]

---

From: Robin Whittle <[email-masked]>
Date: Thu, 29 Nov 2007 23:35:55 +1100

Hi Fred,

Thanks for your response.  You wrote, in part:

> On- and off-list discussions have explored the idea of requiring
> a 2KB EMTU_R on all ETRs and accommodating all 1500-byte and
> smaller packets; even if fragmentation is needed and the inner
> DF=1. This would uphold the "principle of least surprise" to the
> SH, but the issue comes in knowing the EMTU_R of the ETR and in
> avoiding excessive fragmentation.

I don't clearly understand this.  Iljitsch has not been able to
convince me - or anyone else AFAIK - that it it will be practical to
insist on massive hardware upgrades broad enough for the wide range
of location I think ITRs and ETRs need to be located. (Host to host,
Man - Coast to coast. http://www.firstpr.com.au/ip/ivip/tv-ad/)

Maybe some people envisage a more restricted range, or are more
upbeat about the prospects of a global jumboframes and gigabit
Ethernet upgrade.

For the purposes of discussing sprite-mtu and IPTM, I assume that
this upgrade is not practical.

> Since the ITR cannot know the EMTU_R of the ETR a priori unless
> there is some spec that says: "all ETRs MUST configure an EMTU_R
> of at least X bytes", the ITR should not simply fragment the
> outer packets (or, allow the network to fragment them) since they
> could black-hole. 

You and most other people on this list have much more experience in
these matters than I do, but I don't see why fragmenting packets for
a few seconds will lead to a "black hole".

I recognise that fragmentation involves several costs:

1 - More work for the routers at each end.

2 - Two packets rather than one being handled by all
    routers en-route.

3 - More data being sent, due to an extra packet's worth of
    overhead.

4 - Most seriously, I think, a greater chance of loss of the total
    packet due to its delivery depending on two packets, rather than
    one.

    Each packet has some risk of being lost, and the risk is
    a little higher than would otherwise be the case, since the
    two packets take up more time on the wire and more router
    resources than the single original packet.

5 - Fragmentation failures due to 16 bit ID wraparound.  You
    mentioned this being a problem with IPv4 and high packet rates
    with longer delays, causing the wrong packets to be reassembled,
    due to the 16 bit counter  wrapping and them having the same
    sequence number.

Tacking point 5 first, in message 608, quoting Iljitsch, I asked
whether this could be largely resolved by a shorter time window for
reassembly in the ETR:

 - - - -

> I don't have any references, but in short, the issue is that you
> have a 16 bit ID space with a reassembly timeout of something
> like a few minutes. This means you can only send 65536 packets
> during that "few minute" window or you'll incorrectly reassemble
> fragments from different packets if you lose a fragment. This is
> especially problematic if the fragmented packets belong to a
> tunnel because in that case the IP source/dest addresses are
> always the same.

In a fresh system such as an ITR-ETR scheme, perhaps a workaround
for this would be to set the maximum reassembly time at the ETR to
something very much shorter, such as 1, 2 or 3 seconds?

That could be a pain where the ETR is an additional function in a
server or router with a pre-existing TCP/IP stack.  Then it would
probably not be possible to shorten the time just for the ITR to ETR
packets.

 - - - -

Can anyone comment on the prospects of this being an acceptable
solution?

In an ITR-ETR scheme, we probably don't want to sit around waiting
for packets which somehow get lost and arrive more than a second
late via some circuitous route through Manangatang.

I will read:

  IPv4 Reassembly Errors at High Data Rates
  http://tools.ietf.org/html/rfc4963

The other points don't, to me, constitute a "black hole" or a
serious problem.  They seem to involve marginally greater efforts,
marginally less efficiency and marginally greater chances of the
final packet not being delivered.

To me, this is a far better thing to do, for a few seconds or so
into a potentially long flow, than dropping packets.

> There are also other factors to consider, including that the ITR
> may not have ultimate control over the setting of the ip_id. And,
> the ETR may not be able to receive non-initial fragments in the
> first place. (These factors can be mitigated by the placement of
> the ITR and ETR in some use cases, however.)

I don't clearly understand these points.  Can you explain them further?

>> A "long" inner packet is one which, once encapsulated, would
>> exceed the ITR's current best estimate of the PMTU.  This would
>> initially be a default such as 1280 bytes.
>> 
>> If this default value is above the minimum for the protocol,
>> eg. 576 for IPv4, then this value of PMTU to the "core" of the
>> net must be available to every ITR and ETR and would be part of
>> the specification of the ITR-ETR scheme.
> 
> The pathMTU cannot be known a priori; all that can be known a
> priori is the EMTU_R of the ETR if there is a specification for
> the minimum size.

Yes.  I envisage some value around 1280 - at least something so that
a 1500, 1520 etc. byte packet would only be fragmented into two packets.

>> The default value would be replaced by a higher value once the
>> probe process was complete.  The pattern would be something
>> like this, assuming the SH's initial idea of PMTU to the
>> Destination Host was 1500.  I will assume an ENCAPS overhead of
>> 20 bytes (as with IPv4 Ivip, though other ITR-ETR schemes have
>> higher overheads) and that all ITRs and ETRs are located so
>> they have an MTU of at least 1280 from the DFZ.
> 
> Do you mean to say "pathMTU" or "linkMTU"? In terms of "pathMTU",
> do we need to consider links with configurable linkMTUs that
> might have either mis-configured or overly- conservative values?

I don't understand this clearly enough to respond at present.

(My example is not in quotes - just your response.)

                                      1500

             ... etc.

> This could black hole and appear as congestion-related loss to
> the SH.

As noted above, I foresee only marginal efficiency and reliability
impact, where you see unacceptable packet losses.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

> Depending on the placement of the ITR, this PTB might not make it
> back to the SH.

If there is a filter blocking PTB messages from the ITR to the SH,
then I think ordinary, non-RFC4821, PMTUD would be clobbered anyway
- with or without an ITR-ETR scheme  This situation would not
persist, I think when an ITR-ETR scheme was widely deployed, as I
discussed in:

  http://psg.com/lists/rrg/2007/msg00636.html

>> This pattern would continue unless the ITR, with periodic
>> probing, decides that the PMTU is less than 1460 (it might do
>> this quickly if it got a PTB message from a router in a new,
>> more MTU-challenged, path to the ETR), and if the SH sends a
>> packet which would be too big for the new lower value of PMTU.
>> Then the ITR would send another another PTB message to the SH,
>> with a lower value than 1440.

> probes. The data packets will be lost and might be considered by
> the SH as congestion-related loss if the PTB can't be translated
> by the ITR and sent back to the SH. 

In my IPTM proposal, the ITR would only get PTB messages from
routers in the tunnel if the outer source address was that of the
ITR.  This would be the case if IPTM was applied to LISP, eFIT-APT
or TRRP.  Ivip uses the sending host's address in the outer header,
so the ITR would never get a PTB message if the PMTU to the ETR
suddenly became lower than what it had assumed, based on previous
explicit probes acknowledge by the ETR.

This would cause a black hole for all packets which, once
encapsulated, were longer than the new, lower, PMTU between the ITR
and ETR.  To guard against this, the ITR needs to periodically probe
an ETR to which it is continually sending long packets.  There could
be other techniques, such as an ITR-ETR protocol which enables the
ITR to receive acknowledgement of packets it sends, but this gets
pretty complex, and I am hoping to avoid such things.

> But, the ITR will be able to return the correct PTB when the SH
> retransmits. This is the same as for sprite-mtu.

Once the ITR has a correct idea of the new, lower, PMTU to the ETR,
it can drop (IPTM) or send (sprite-mtu) the packet, and generate a
PTB to the SH when the SH next sends a packet which, once
encapsulated, would be too long for that new PMTU.

>> Alternatively, occasional probing by the ITR might discover a
>> higher value of PMTU to this ETR, and the SH could discover
>> this increase by trying its luck with a larger packet - and
>> either having it accepted, or rejected with a PTB containing
>> the new higher value, minus 20.
> 
> SHs that don't implement RFC4821 will have to wait for a long
> time before trying a larger packet (RFCs 1191 and 1981 say 10min,
> I believe). 

Indeed.  This is why my critique of sprite-mtu seems to be
important.  As I understand it, the SH first gets a rather low value
in the PTB message from the ITR, at a part of the exchange at which
an IPTM ITR would have simply fragmented the packet without any PTB,
and commenced or continued probing the ETR.

If it takes at least 10 minutes for a non-RFC4821 compliant host to
try sending larger packets, then this is a long time for the
communication to be restricted to the shorter packets.

I am not assuming widespread adoption of RFC4821 at any time.  It
looks really complex to implement, involving applications and the
TCP layer communicating with a new function in the OS in ways which
were not originally part of the protocol stack.   Writing all this
code, for marginal immediate benefit, and then trying to debug it in
all its possible combinations of applications, live network settings
etc. sounds really, really, complex.

> SHs that implement RFC4821 can retry end-to-end probing more 
> frequently than that, since loss of a probe does not
> expose data to silent loss.

Yes.

>> irrespective of the DF flag of the inner packet.

OK.

>> The criteria for which IPv4 outer packets are fragmentable is
>> complex (5.6.4).
> 
> discipline set forth in RFC4213, Section 3.2. But, I don't know
> what you mean by complex?

What I meant to write was that when meant I tried to read this and
the rest of the ID in order to answer the questions I tried to pose
in my second example, I couldn't understand it clearly enough to
envisage how your proposal would operate.

My questions were something like:

  Does the ITR first send a rather low value in the PTB message (for
  the first "long" packet)?

  While probing is taking place, does the ITR generate PTBs for
  packets which are too long for its currently too low PMTU
  estimate (and sends the packet too, without fragmentation), or
  does it fragment them, like IPTM?

I think the answers are:

  Yes.

  The ITR generates PTBs and sends the packets holus bolus.

>> I am not sure how Sprite handles "large" outer packets while it
>> is probing.  Does it fragment them as IPTM does?  Or does it do
>> the following, which is the same as what it does for a "long"
>> packet once the PMTU has been reliably ascertained:
>> 

OK, without carefully reading RFC2003, which I would if I had more
time right now, I understand that the ITR both sends the complete
outer packet, without fragmentation, and sends a PTB message to the SH.

>> It seems strange to me to send the packet (unfragmented, I
>> assume) while also sending back a PTB message to the sending
>> host.  Wouldn't this cause needless traffic and/or confusing
>> signals to the SH if the outer packet does in fact arrive at
>> the ETR and therefore the inner packet is delivered to the
>> destination host?
> 
> To the SH, it would appear that there is a router on the path
> returning inaccurate information. This can happen already today,
> since routers can be misconfigured, and spoofed PTBs can be sent
> from any node in the network.

It still seems strange, confusing and inefficient to me.

> end-to-end feedback from the DH, but should benefit from the PTB
> info when the actual data is not delivered to the DH.

Yes, but I am assuming that none, or few, hosts will implement
RF4821 any time soon.

> ITRs can help the situation by sending sprites of, e.g., 1500
> bytes into the tunnel early in the process so that most if not
> all SHs that use the tunnel will see a 1500 byte or larger MTU.

Does "early in the process" mean when only shorter packets have so
far needed to be tunneled to the ETR?

If so, then the ITRs could be generating large volumes (in bytes) of
probe packets in response to only small traffic flows, and to some
flows which never in fact require PMTU knowledge, since the flows
never actually use long packets.

>> Here I will assume IPv4 only, with 1280 bytes for the default
>> PMTU for every ETR the ITR has not yet probed.  I will also
>> assume an encapsulation overhead of 20, although this would
>> typically be higher for Sprite and non-Ivip ITR-ETR schemes.
> 

This was a low-key aside.  In my second example, trying to explain
how I thought sprite-mtu might work, I kept the same 20 byte
encapsulation overhead I used in my first example, which was for
IPTM, assuming Ivip's 20 byte overhead.

Other ITR-ETR schemes, and I guess most other tunneling schemes
sprite-mtu would be applied to, have higher encapsulation overheads,
I think.

>> If the ITR sends a PTB message to the SH when the first packet
>> (or multiple packets) length exceeds the default PMTU value and
>> then, after probing, decides the PMTU is 1480, then I am
>> concerned that the SH would get contradictory values in these
>> PTB messages.
>>
>> At first the SH would be told to send packets no longer than
>> (1280 - 20 = 1260) and later, it would be told to send packets
>> no longer than (1480 - 20 = 1460).
>
> Note: in the next draft version I would like to rewrite the
> second bullet of Section 5.6.4 as:
>
> o  for IPv4/*/IPv4 tunnels, 'pathMTU' is less than MIN(EMTU_R,
>    1280+ENCAPS) bytes and the inner IPv4 packet is no larger than
>    MIN(EMTU_R-ENCAPS, 1280).

The current version is:

  o  for IPv4/*/IPv4 tunnels, 'pathMTU' is less than MIN(EMTU_R,
     1280) bytes and the inner IPv4 packet is no larger than
     MIN(EMTU_R, 1280) minus ENCAPS.  (When EMTU_R for the TFE is
     not known, 576 bytes must be assumed.)

OK.  My eyes are glazing over right now.  I will probably be able
to understand this in its full context in the future.

>> wouldn't send any longer packets which would trigger the
>> second. So, if my understanding of Sprite is correct, the SH
>> would experience something like this:

                                      1500

> Probing can start here also.

As I noted above, at this point the ITR can't reasonably assume that
it will need PMTU information to the ETR.  It is just a short packet.

Perhaps smart ITRs could have heuristics to help them decide when to
start probing earlier, based on prior statistics (of all sorts of
aspects of packets that the ITR might analyse) of shorter packets
often being followed by longer packets.  But that is not something I
would require of an IPTM ITR.

> With the note above, the returned size would be 1280; not 1260.
> Also, I'm not sure as to the "probably nothing" as linkMTUs
> increase above 1500 (perhaps someone could send the IEEE
> reference that proposes the increase for 802 linkMTUs).

OK - I guess in the future it might be worth trying longer than 1500
byte packets to the ETR.

My current IPTM proposal is not aimed at optimising the
opportunities for doing so.  I will think about this whenever I
revise it.

             SH breaks the message
             into smaller packets
             and retries:

             ... etc.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

> By probing, do you mean by the ITR or by the SH? 

I meant the ITR sends sprite probes to the ETR.

> I am assuming that SHs will begin using RFC4821 and will
> probe the path for themselves independent of any probing
> done by the ITR.

I am not assuming hosts will be any different than they are today.
As far as I know, few, if any, implement RFC4821.

If you are assuming this, I think it would be good to make it an
explicit condition you are designing sprite-mtu to function within.

> IMHO, SHs that use RFC4821 can be "pushy" within reason.

Yes, but I think it will be a long time before there are many such
hosts.

> I would like to add one other note about the 1280. That number
> comes from the SHOULD in RFC4213, Section 3.2.1. The reason I am
> taking the SHOULD is that there can be additional encapsulations
> on the path between the ITR and ETR (e.g., tunnel-mode IPsec) and
> we don't want to cause fragmentation for those, either. If the
> ITR and ETR are arranged such that there will be no additional 
> encapsulations on the path (and the ITR has a way of knowing
> this) then the spec could use the RFC4213, Section 3.2.1
> "configuration knob" to push the 1280 up to as much as 1480 or
> perhaps even 1500. I would not want to go any higher than this,
> since it could involve excessive fragmentation resulting in
> undetected data corruption. 

OK - I won't try to chase this up right now, but thanks for the
pointers.

> Maybe I should add something about this to the spec?

Yes, I think the more explanation of where numbers like 1280 come
from, the better.

Thanks again for this detailed discussion.

  - Robin     http://www.firstpr.com.au/ip/ivip/pmtud-frag/

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 29 Nov 2007 09:13:09 -0800

Robin,

See below: 

> Sent: Thursday, November 29, 2007 4:36 AM
> Subject: Re: [RRG] Sprite & IPTM while PMTU probing is in progress
> 
> Hi Fred,
> 

Sorry for asking, but do you have a PR department that
writes this stuff up? :^}
 
> this upgrade is not practical.

IMHO, the H/W upgrades will occur independently of any
tunnel MTU handling. But, the tunnel MTU handling should
be capable of taking advantage of the larger MTUs when
they become available.
 
> a few seconds will lead to a "black hole".

If the ETR does not have a large-enough EMTU_R, I think
it would have to drop the packet silently.

> 1 - More work for the routers at each end.

That brings up another question; were you expecting to
fragment the packet in the ITR's stack before sending,
or just send with DF=0 and let the network fragment?
The literature and recent discussion consensus seems
to greatly favor the former.
 
> 2 - Two packets rather than one being handled by all

Or more than 2, if the network also fragments.
 
> Can anyone comment on the prospects of this being an acceptable

During some of the discussions (can't recall whether it
was on- or off-list) some Linux code was shown that uses an
out-of-order upper bound in terms of number of reassemblies
ourstanding instead of a shortened timer when deciding whether
to purge an incomplete reassembly. That seemed like the right
way to go to me.

Setting a shorter timer might not be such a good idea for
reasons outlined in RFC1122, Section 3.3.2. In some earlier
efforts, I delved into the RFC1122 suggestion of managing
round-trip times: 

  + It has been suggested that a cache might be kept of
  + round-trip times measured by transport protocols for
  + various destinations, and that these values might be used
  + to dynamically determine a reasonable reassembly timeout
  + value.  Further investigation of this approach is
  + required.

My conclusion was that a protocol between the ITR and ETR
could be devised to do this, but it was far too complicated
for practical purposes and had too many things that could
go wrong.
  
> late via some circuitous route through Manangatang.

There is also the consideration of whether we want to make
this delay-tolerant, e.g., for multiple satellite hops,
interplanetary communications, etc.

OK.

> I don't clearly understand these points.  Can you explain 

If the ITR is located behind a NAT or translating firewall,
the ip_id could be re-written. I believe a common approach
is to rewrite with a randomly-chosen 16-bit value. When
there are many SH's behind the same NAT/firewall and using
the same ITR, this could present a problem.

About non-initial fragments, I am told that some NATs/firewalls
simply pass the first fragment and drop all others. Problem
being that non-initial fragments do not include the transport
layer header.
 
> a 1500, 1520 etc. byte packet would only be fragmented into 

Unless some new spec comes along, the only values for
EMTU_R that can be assumed are 576 bytes for IPv4 and
1500 bytes for IPv6.

> I don't understand this clearly enough to respond at present.

May not have been worded very well. A while back, I think
it was Iljitsch who indirectly pointed out that any link
for which an MTU can be manually configured can also be
misconfigured. For example, what if an operator means to
set a linkMTU of 4500, but his finger slips and he ends
up setting only 450? Flows that use that link would
experience the inefficiency - but, they should still work. 

> impact, where you see unacceptable packet losses.

It's not so much the loss I am concerned with; IMHO,
sustained and unmitigated fragmentation is dangerous
even for short periods of time.

>   http://psg.com/lists/rrg/2007/msg00636.html

See RFC2923 for a discussion of PTB filtering implications
for classical PMTUD. 
 
> explicit probes acknowledge by the ETR.

I believe this represents a departure from some of the other
proposals. Are you suggesting that the SH will always be
globally addressable from within the core? So, there is no
locator/id split? No public addressing for the ITR and
private addressing for the SH? No IP version mismatches
between the SH and ITR?

In any event, assuming that the outer source is that of
the SH (and not the ITR), then the PTBs coming from within
the tunnel and delivered directly to the SH will forever
report a too-large MTU because the SH has no way of knowing
that an ITR on the path will be inserting ENCAPS overhead.
Even worse, the data included in the PTB will include the
ENCAPS inserted by the ITR and will be unrecognizable
to the SH.

> pretty complex, and I am hoping to avoid such things.

Unless the period for probing is very short, there is
opportunity for a self-sustaining denial-of-service to
the SH. 
 
> encapsulated, would be too long for that new PMTU.

This loses out on the opportunity to use the packets
sent into the tunnel as probes to detect MTU increases.
And, if you are expecting your ITR to periodically
re-probe to detect MTU increases in time to satisfy
MTU-probing SH, it is going to require excessive probing
overhead and may not get the job done in time to avoid
black-holing. 
 
> and commenced or continued probing the ETR.

That depends on what you consider to be "low". Depending
on the placement of the ITR, "low" will only go as low
as 1280 but may go as large as, e.g., 1480. 
 
> communication to be restricted to the shorter packets.

SHs are therefore advised to begin implementing RFC4821.
Deployment is incremental and involves only the SH.

> etc. sounds really, really, complex.

I disagree; active end-to-end involvement in MTU determination
is important for the long term.
 
> It still seems strange, confusing and inefficient to me.

I disagree; there is value in sending the packet into the
tunnel on at least three levels: 1) it serves as a virtual
probe so that the ITR can detect MTU restrictions further
down the tunnel, 2) the packet may be an MTU probe of the
SH, 3) Packet delivery ratio may benefit in some use cases. 
 
> Yes, but I am assuming that none, or few, hosts will implement

Incrementally deployable; touches SH only; realizing
larger MTUs gives incentive for deployment.
 
> never actually use long packets.

There is nothing mandated here, and implementations will
be evaluated on the merits of their probing strategies. 
 
> sprite-mtu would be applied to, have higher encapsulation overheads,

The ENCAPS overhead is orthogonal to the use- or non-use
of sprite-mtu.

> OK.  My eyes are glazing over right now.

Reaching that point also myself...

> I will probably be able

Again, nothing is mandated and some implementations may
make have better probing strategies than others.

> OK - I guess in the future it might be worth trying longer than 1500

Right.
 
> I meant the ITR sends sprite probes to the ETR.

Do you mean sprite-mtu probes and not traceroutes? One
thing I do not understand about your proposal is how
you expect the traceroutes to be efficient and converge
within a reasonable amount of time? Also, there is no
guarantee of getting the ICMPs back from the network
middleboxes. Am I missing something?

> As far as I know, few, if any, implement RFC4821.

If hosts see the value of larger MTUs, they will begin
to deploy RFC4821. (Or, if vendors see the value, they
will begin to push out RFC4821 in automated S/W updates.)
IMHO, PMTUD cannot be efficiently and correctly handled
within network middleboxes alone; end-to-end involvement
is needed as well.

> explicit condition you are designing sprite-mtu to function within.

sprite-mtu works independently of RFC4821, but end systems
benefit from using RFC4821.

> Yes, but I think it will be a long time before there are many such

Why not?

OK.
 
> Thanks again for this detailed discussion.

Sure.

Fred

---

From: Robin Whittle <[email-masked]>
Date: Fri, 30 Nov 2007 15:58:36 +1100

Hi Fred,

> writes this stuff up? :^}

It is more a case of adding some colour to keep me awake.
Routing and addressing is pretty dry, I think.  The LISP folks
may have had the same experience - it seems they were dreaming
of Iphones.

> be capable of taking advantage of the larger MTUs when

I agree.  I will try to ensure that IPTM can make use of much
larger MTUs.

> it would have to drop the packet silently.

Yes, but this doesn't seem pertinent to my critique of sprite-mtu
that it would be better to:

1 - As IPTM does, fragment the outer packet into two smaller
    ones, which will pass through the tunnel OK.  Hmm, this
    embodies an assumption that the outer packet is no bigger
    than 1500 bytes or so, and that all ITRs and ETRs are to
    be located in places where they have at least 1280 (or
    similar) byte MTUs to the DFZ.

2 - As sprite-mtu does, try sending the entire outer packet
    (assuming it fits in the MTU limit of the next hop)
    while also sending a PTB to the SH, with a value
    designed to ensure the SH sends only packets which, once
    encapsulated, will fit within the ITR's current estimate
    of the PMTU to the ETR.  Initially, this estimate is a
    low default value, since the ETR probing is not yet
    complete.

I have been focusing on hosts which assume they can send
packets up to 1500 bytes long, and trying to make the ITR-ETR
system handle these, initially, with fragmentation, but then
use a PTB (or as many as are required before the SH changes
its ways) to get the SH to send somewhat smaller packets.

I need to think more about the future, where hosts might
send really large packets, many times bigger than 1500 bytes,
while the ITR still has to assume a figure of ~1280 for the
MTU to an ETR it hasn't yet probed.

When fragmentation means many more than 2 fragments, your
objection to fragmentation make more sense to me.

Here I will consider jumboframes to be 16114 bytes, according
to Iljitsch's: http://psg.com/lists/rrg/2007/msg00628.html .
Jumboframes could be longer, which is more difficult still.

With ITPM as it stands and a jumboframe-inclined SH, which
sends ~16114 byte packets, the ITR would fragment it into 11
packets.  This greatly increases inefficiencies and the chance
of one being lost - assuming the PMTU to the ETR, and the ETR
itself, was ready for (16114 + ENCAPS) byte packets.

My primary goal is to handle existing hosts - which are assumed
generally not to be jumbo-inclined, and which I guess generally
put out packets of ~1500 bytes, according to their local link
MTUs - in an environment with widespread ITR-ETR use, over links
which may have slightly sub-1500 MTUs themselves, causing the
ITR-ETR tunneling overhead to limit the payload size to some
value ~20 to ~100 bytes below 1500.

I think IPTM does a better job of this than sprite-mtu - because
I think IPTM's fragmenting of larger packets into two is a better
course of action than sending a PTB (and also sending the single
large outer packet).

A secondary goal - which will be really important in the long
run - is to craft IPTM so it works well in a world where
many links, but not all, handle jumboframes, and where hosts
tend to send jumboframes.

One approach is to do something to IPTM in a fixed way to
create happy outcomes in both scenarios.  Another is to
make some requirement of hosts in the future which are
inclined to send jumboframes - I would rather avoid this.

Another is to plan for some overall change to the behavior
of IPTM devices in the future.  This is assuming that there
isn't a single way of optimising for both both scenarios,
so at some point, when it is worthwhile, we switch over to
an alternative mode of operation, different config settings
etc. which is worse for the old scenario, but better for
the jumboframes scenario.

> to greatly favor the former.

Hmm ... I was intending the ITR to fragment the packet.

This was to avoid a situation where there are two or more
MTU bottlenecks in the path:

  ITR sends 1 x 6000 byte packet.

  R1 (seeing MTU to R2 is 3000 bytes) fragments this to
  3 x 2020 ((6000 / 3) + 20) byte packets.

  R2 (seeing MTU to R3 is 1500 bytes) fragments these to
  6 x 1030 ((2220 / 2) + 20) byte packets.

IPTM would have assumed a ~1280 byte PMTU, until it had
completed probing the ETR, so it would have fragmented
the packet to 5 x 1220 byte packets ((6000 / 5) + 20).

This is marginally more efficient than the above, except
for where the single packet can travel most of the distance
before fragmentation.

Thanks for pointing this out.

> Or more than 2, if the network also fragments.

Yes, but for packets which start off around 1500 bytes, and if
we make a "1280 PMTU to the DFZ" rule for placing all ITRs and
ETRs, we can assume that the packets will only be fragmented
once into two.

> to purge an incomplete reassembly. That seemed like the right

OK.  This sounds easier to implement than timers.  Hopefully
someone can rememember some information about this.

> for practical purposes and had too many things that could

It is just the ETR which has to be changed.  This raises
questions of whether the ETR is some special new device
(unlikely) or a conventional PC-like host, which would
typically rely on standard OS packet reassembly, which
could be hard to alter.  A server specifically programmed
to be an ETR might have tweaked networking code in is OS
for this purpose.

Routers may or may not be amenable to tweaking their packet
reassembly operations just because the router is functioning,
in part, as an ETR.

> the same ITR, this could present a problem.

In Ivip, ITRs are never behind NAT.  I assume the same is
true of other ITR-ETR schemes.

> being that non-initial fragments do not include the transport

I don't fully understand this, but I think it doesn't matter
if we know ITRs are never behind NAT etc.

> EMTU_R that can be assumed are 576 bytes for IPv4 and

The new specification would be part of the ITR-ETR scheme's RFC.
For instance:

   ITRs and ETRs must be located so that their PMTU to the
   DFZ is at least 1280 bytes.

The number would need to be carefully chosen.  I use 1280
because it is substantially below 1500 - and so able to
accomodate several layers of tunneling.

The trick would be to make the number reasonably high, to
reduce the number of initial packets the IPTM ITR would need
to fragment, or worry about at all in terms of PMTU, without
unnecessarily restricting the location of ITRs and ETRs.

> experience the inefficiency - but, they should still work. 

I don't think a major architectural addition like an ITR-ETR
scheme with its sprite-mtu or IPTM or whatever PMTU management
system should be required to cope gracefully with such
misconfiguation.

> even for short periods of time.

My feeling - and you have much more experience in this than I do -
is that short bursts of fragmentation into two packets is a better
choice than instantly telling the sending host to send much shorter
packets than it probably can send.

But this is with my assumption the SH is only putting out packets
about 1550 bytes long.  It doesn't hold if hosts are pushing out
jumboframe packets over 2000 bytes or so.

> See RFC2923 for a discussion of PTB filtering implications

I think I have read this, but I will read it again.

> proposals. 

It is, for reasons including:

  1 - Making it relatively easy for the ETR to enforce the
      ISP's filtering which rejects incoming packets with
      source addresses from its own network - extending
      this protection to the inner packet, by refusing to
      forward to the destination host any packet where the
      inner source address is different from the outer
      source address.

         http://tools.ietf.org/html/draft-whittle-ivip-arch-00#section-14.1

  2 - Enabling the SH to perform traceroute, with a somewhat
      modified traceroute program.

but this choice of outer source address = sending host address is
also tied in with some other decisions in which Ivip is different
from the other proposals to date.

> Are you suggesting that the SH will always be

The SH may be behind NAT, but since the ITR is betwen the NAT and
the DFZ, the ITR regards the SH's address as that of the NAT box.

SHs can be on ordinary addresses, receiving packets directly via
the current BGP system, or they can be on Ivip-mapped addresses,
receiving packets via ITRs and an ETR.  In Ivip, the mapped
address blocks are all advertised in BGP, but with an "anycast"
arrangement where multiple (hundreds of thousands, for instance)
ITRDs (full database ITRs) advertise each mapped address block,
causing packets from sending hosts to go to the nearest such ITRD.

I don't think of Ivip as a locator-ID separation protocol.  It
probably is, but I see it as more of an additional system of
plumbing - to do a much finer, and more rapidly flexible in time,
job of splitting up the address space and getting packets to
ETRs near the destination hosts than is practical with BGP.

> ENCAPS inserted by the ITR and will be unrecognizable

Yes, so a correctly implemented SH should will ignore them - so
these PTBs will cause no trouble.

The SH needs to get a PTB from the ITR, which is what IPTM
is intended to achieve.

> opportunity for a self-sustaining denial-of-service to

I don't see how there could be a DoS attack as such.

Unless I add something to IPTM to have the ITR receive
acknowledgements, say every 10 to 60 seconds, from the ETR,
then the ITR has no way of knowing that the tunnel is
dropping packets, due to the PMTU to the ETR becoming
lower than the value discovered during probing.

Another approach is to have more frequent ITR probing of
the ETR, as you suggest.

I intend IPTM to be usable by other schemes than Ivip.  For
those schemes, the IPTM ITR would usually be able to detect
that the tunnel PMTU had been reduced, since the ITR would
get the PTB messages from one of the routers in the tunnel.

However, this is not robust, because there could be a filter
between the ETR and ITR which might prevent the PTB packets
from reaching the ITR.

IPTM for Ivip doesn't care about any filtering of PTB
messages between the ITR and the ETR.  It relies entirely
on explicit probes from the ITR to the ETR.

> overhead and may not get the job done in time to avoid

I am not too fussed about how rapidly the ITR detects an
increase in PMTU to the ETR.  Delay in discovering this only
means a period of lost opportunity for increased efficiency.

Delay in discovering a shorter PMTU is a much more serious
problem, because that leads to black holes of complete
packet loss.

Of course, if RFC4821 is widely implemented, the SH will
discover the packet loss promptly and try shorter packets.

I think RFC4821 is basically a good thing.  IPTM doesn't
get in its way, and in this particular packet loss situation
which IPTM as currently defined would create, RFC4821 would
make this limitation of IPTM not much of a problem.

However, IPTM doesn't require or (I think) otherwise benefit
from RFC4821.

> as 1280 but may go as large as, e.g., 1480. 

Ignoring for the moment IPTM's current difficulty in coping
nicely with SHs and tunnel paths which handle jumboframes, my
critique of sprite-mtu is one of the relative efficiency,
between, for instance:

 1 - IPTM fragmenting the first long packets while it probes
     the ETR, and then settling down to a nice ca. 1460
     packet length from the SH, after the ITR sends a PTB
     message to it.

 2 - Sprite-mtu, as I understand it, first sending a PTB of
     ca. 1260, and the SH using this for the next 10 minutes
     before it is allowed to try sending a longer packet.

How high can we make this ~1280 default figure?  The higher
the better, as long as we don't unnecessarily restrict the
placement of ITRs and ETRs.

However, since many or most flows of information last less
than 10 minutes, and since most SHs (in my view) will not
be using RFC4821, this means that with sprite-mtu, most
traffic would be stuck with the default value MTU, rather
than whatever higher value the ITR discovers with a few
seconds of probing.

I think sprite-mtu only works well if the SH uses RFC4821.

An RFC4821 SH won't be negatively affected by the ITR both
sending it a PTB message (with a PMTU value ca. 1260 or so,
as per my example - or perhaps (576 - ENCAPS) because
if the packet it sent was not too long for the tunnel (once
encapsulated) then it would get end-to-end RFC4821
confirmation of delivery, and so ignore the PTB message.

If it took notice of the initial PTB message(s), an RFC4821
SH is still able to fight its way out of this inefficient
situation (sending packets of ~1260 or less bytes) by
trying a longer packet length in a shorter time than allowed
for non RFC4821 compliant hosts.

> Deployment is incremental and involves only the SH.

I think this makes the acceptable behavior of the ITR-ETR
scheme (at least in terms of solving the PMTUD problem
which bedevils every such scheme) dependent on host changes.

This is at odds with my vision of incremental deployability.

I wrote about why I think RFC4821 is a very demanding host
change which I think is unlikely to be widely adopted in
the time frame in which an ITR-ETR scheme needs to be
introduced, which I see as being within 5 years.

Does anyone argue that RFC4821 adoption in desktops and
servers, are actually going to be widespread in the next
5 or 10 years?  I know it is a new RFC, but what is the
current status of implementation, in operating systems
(TCP) and in applications?  Applications can only do it
if the OS supports it.

> is important for the long term.

I agree it is important in the long term, but is it
really going to happen in the next few years?

Sufficiently for it to be widely enough deployed that
those SHs which don't have it and therefore don't work
well with sprite-mtu, will experience persistent PMTU
difficulties as a result of  the introduction of the
ITR-ETR scheme?

If there are a significant number of hosts (maybe as
few as 10% or less) which are not up-to-speed with
RFC4821, in the early days of introducing an ITR-ETR
scheme (I hope 2010 or 2011), then I fear this would
have the the effect of making addresses mapped by
the ITR-ETR scheme suck - which would create an
impenetrable barrier to the introduction of the scheme,
and therefore doom our beloved Internet to eternal
twilight and doom.  (Or boost the fortunes of Cisco
et al. as everyone buys new routers to cope with
millions of DFZ routes - the demand for IPv4 space
in smaller increments is sure to drive growth in
advertised prefixes up precipitously in the next
5 to 10 years.)

...

> SH, 3) Packet delivery ratio may benefit in some use cases. 

I can see this value, although IPTM at present has no way of
using encapsualated traffic packets as probes.  I would have
to add an ITR-ETR protocol for that to be available - and
that would involve either adding a special header or trailer
to the encapsulated packet (so the ETR could uniquely
identify it), or devising some robust scheme by which the ETR
could tell the ITR exactly which packets it received, based
entirely on the naturally occuring characteristics of the
encapsulated packets.  Either option involves the ITR in
a bunch of record keeping and communications guff with the
ETR - and I am trying to keep this really lightweight.

> be evaluated on the merits of their probing strategies. 

I think an RFC for sprite-mtu or IPTM should give some guidance
on when to probe, but allow the ITR to make its own decisions
and/or be configured to suit local conditions.

> The ENCAPS overhead is orthogonal to the use- or non-use

I was just explaining my continuance of 20 as the ENCAPs value
from one example to the next.  However, I understand that
sprite-mtu does, however, involve extra trailers on some
traffic packets.

> Reaching that point also myself...

Indeed.  Is anyone else keeping up with this epic correspondence?

> middleboxes. Am I missing something?

This example is to explore my understanding of sprite-mtu.

Above, I mean that the ITR uses special probe packets to the
ETR, called "sprites".  I don't recall traceroute being a
part of your proposal.

In IPTM, the ITRs or ETRs don't do traceroute.  With Ivip's
"outer source address = sending host address" approach,
I propose that a modified traceroute program should be able
to trace all routers, including those in the tunnel, assuming
the ITR replicates the TTL value when encapsulating.  This
would involve moderate changes to the traceroute code to
recognise the ICMP packets which come back from the tunnel,
which are for the outer packet, not the inner as sent by
the traceroute program.  Such a traceroute program would be
able to depict which routers were in the tunnel, and also
determine the ETR address, from the ICMP messages coming
back from routers in the tunnel.  I have only looked at
this quickly, so perhaps there are problems with this
proposal.

Enabling traceroute from the SH, all the way through
the ITR-ETR tunnel, is not possible with other ITR-ETR
schemes unless the ITR performs heroics which I think are
prohibitively demanding.  Traceroute is a valuable
debugging tool and as far as I know (someone confirmed
this on-list) no applications rely on traceroute.
administrative and debugging benefit

> within network middleboxes alone; end-to-end involvement

As noted above, I think it is a great idea, but it is
very complex and requires applications and OS to work
together.  So the OS framework and TCP stuff needs to be
done before an application could be made to work with it.

I have no idea to what extent this is being done, but it
sounds very complex and likely to happen slowly at best.

> benefit from using RFC4821.

But without RFC4821 in the SH, as far as I know, the SH
will be constrained for 10 minutes at least to the PMTU
value it gets in the first PTB message sent by the ITR.

This is for a lower value than the ITR would send after a
few seconds probing.

So I think an RFC4821 compliant SH is necessary for your
system to deliver communications with packet lengths
longer than the low, default, value the ITR must assume
at first.

If my understanding is correct, I think it would be good
to state this in some way in later versions of your ID.

>> Yes, but I think it will be a long time before there are many such

I already discussed the complex software requirements for
implementing RFC4821 in hosts, including the establishemnt of
new paths of two-way communication between applications and
whatever part of the OS network code is responsible for
RFC4821.

I think that requires new standards in OS calls, and I guess
ideally some OS independent way of writing applications
which handles this stuff.

>> Yes, I think the more explanation of where numbers like 1280 come

  Cheers

    - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 30 Nov 2007 09:22:51 -0800

Robin,

I am going to top-post and hit what I think are the high
points; please let me know if I have missed anything:

1) My primary concern with what you are describing is that
by setting the source address of the tunneled packet to that
of the SH (and not that of the ITR) you are effectively
disabling all ICMP error messages (not just the PTBs) for
both the SH and the ITR. The ITR will never get ICMP feedback
from routers within the tunnel, and the SH will get ICMP
feedback that it does not recognize. The lack of ICMP feedback
may have negative implications from a protocol perspective
for both the ITR and the SH. And, I do not know this for sure,
but I believe many if not most SH's will view the unrecognizable
ICMPs as an attack, e.g., warning bells may go off at least,
and more onerous defense mechanisms may kick in in the extreme.
This concern may extend to Ivip in general, and not just the
pathMTU handling.

2) Secondly, while there are routers within the tunnel that
are sending meaningless PTBs (since neither the SH nor ITR
will get them) the tunnel will "brown"-hole. You are not
going to be able to probe the ETR frequently enough to
avoid periods of either silent loss or unmitigated
fragmentation depending on the setting of the DF bit in the
outer IP header. If you tried, the probing overhead would
be excessive and then you still would never be able to probe
frequently enough to avoid all brown-outs (I classify
unmitigated fragmentation as a brown-out even if the
fragments arrive at the ETR).

3) You have not characterized the expected behavior of the
(modified) traceroute used for probing. If there is a path
of N routers, then an unmodified traceroute would need to
send up to N big packets to determine whether they are
reaching the ETR (and, the loss of a big packet at any hop
would result in a significant delay to detect). Also, raw
pings don't work because sending a large echo request causes
an equally-large echo reply to come back - and, the pathMTU
is not necessarily symmetric in the forward and reverse
directions. 

4) Both sprite-mtu and your approach operate in the
presence of fragmentation, but sprite-mtu explicitly
manages fragmentation while yours just lets it fragment.
Sprite-mtu could include a "configuration knob" (per
RFC4213) to set a higher fragmentation threshold than
1280 based on knowledge of the ETR's EMTU_R, but this
would be used at the peril of reassembly misassociations
and potential data corruption if no mitigations were in
place. (BTW, the sprite-mtu trailer is added only when
the tunnel is fragmenting and is there for the purpose
of detecting packet splicing errors.)

Again, please let me know if I missed covering any of
the points you believe are important.

Fred

> Sent: Thursday, November 29, 2007 8:59 PM

---

From: Robin Whittle <[email-masked]>
Date: Sat, 01 Dec 2007 19:56:59 +1100

Hi Fred,

Thanks very much for this constructive critique.

I have linked to the messages of this discussion from my page:

and will revisit these things in the future.

  Cheers

    - Robin