Subject: [rrg] RANGER and SEAL critique

From: Robin Whittle <[email-masked]>
Date: Tue, 26 Jan 2010 22:48:18 +1100

A word file of the following text is attached.

This is my attempt to write a critique of RANGER and SEAL for the
RRG Report.  The SEAL critique is at the end.

Since there is a 500 word limit for the RRG report, I suggest
that just the first 7 paragraphs be used.  Trying to discuss
RANGER in any more detail quickly expands to much larger task.

This is not an exhaustive critique.  I have done enough to
satisfy myself that RANGER, as currently specified, should not
be considered as a possible practical solution to the routing
scaling problem.

This is just my writing - it does not reflect RRG consensus and
it has not benefited from any input from Fred or anyone else.
Please watch the RRG list for any clarifications, corrections etc.

Maybe someone else can improve on this critique.

Robin Whittle  2010-01-26

Critique of Fred Templin's RANGER and SEAL

The RANGER architectural framework is intended to be applicable
for a Core-Edge Separation (CES) architecture for scalable
routing, using either IPv4 or IPv6 - or using both in an
integrated system which may carry one protocol over the other.

However, despite the ID being readied for publication as an
experimental RFC, the framework falls well short of the level of
detail required to envisage how it could be used to implement a
practical scalable routing solution.  For instance, the ID
contains no specification for a mapping protocol, how the
mapping lookup system would work on a global scale.

There is no provision for RANGER's ITR-like routers being able
to probe the reachability of end-user networks via multiple
ETR-like routers - nor for any other approach to multihoming
service restoration.

Nor is there any provision for inbound TE or support of mobile
devices which frequently change their point of attachment.

Therefore, it its current form, RANGER cannot be contemplated as
a superior scalable routing solution to some other proposals
which are specified in sufficient detail and which appear to be
feasible.

RANGER uses its own tunneling and PMTUD management protocol:
SEAL.  Adoption of SEAL in its current form would prevent the
proper utilization of jumboframe paths in the DFZ, which will
become the norm in the future.  SEAL uses RFC 1191 PTB messages
to the sending host only to fix a preset maximum packet length.
 To avoid the need for the SEAL layer to fragment packets of
this length, this MTU value (for the input of the tunnel) needs
to be set significantly below 1500 bytes, assuming the typically
~1500 byte MTU values for paths across the DFZ today.  In order
to avoid this excessive fragmentation, this value could only be
raised to a ~9k byte value at some time in the future where
essentially all paths between ITRs and ETRs were jumboframe capable.

A fuller version of this critique was posted to the RRG list on
2010-01-26.

<<< - - - - End of <500 word critique - - - - - - - - - - - ->>>

RANGER involves novel and potentially confusing terminology and
is explained in terms of the greatest generality - which makes
it difficult to determine the simplest arrangement necessary for
widespread adoption in order to solve the routing scaling problem.

Encapsulation is used to tunnel packets from an ITR like device,
which may be referred to as an "Ingress Tunnel Endpoint" (ITE)
or "ingress Enterprise Border Router" (iEBR) to an ETR-like
device - an "Egress Tunnel Endpoint" (ETE) or an "egress
Enterprise Border Router" (eEBR).

The term "enterprise" can refer to a network of any size,
including the Internet.  SEAL's extendibility goes beyond this:
"other internets could be manifested as 'parallel universes' and
joined together at still higher levels of recursion".

A further difficulty is that RANGER uses the term "RLOC" with
greater flexibility than any other CES architecture. If the
subset of the global unicast space of the Internet which is not
devoted to EID (scalable end-user network) space is regarded as
"RLOC" commons for the enterprise of the Internet, then within
the RANGER model, any subsidiary enterprise which connects to
this Internet enterprise has its own RLOC commons, which may (or
must?) involve independent re-use of the RLOC space of the
Internet, plus any private address space which this subsidiary
enterprise uses.

It is different to envisage scalable routing with full
interworking to hosts in other networks using "Internet RLOC"
addresses from a subsidiary enterprise network if its own use of
these addresses is wholly, or partially, or at least potentially
"private" to this enterprise and therefore in a separate namespace.

While RANGER is claimed to be suitable for both IPv4 and IPv6,
some of its internal mechanisms depend on IPv6.  So it is
especially difficult to envisage a pure IPv4 implementation of
RANGER to solve the IPv4 scalable routing problem.  The ID tends
to regard IPv4 communications as "legacy".

Although RANGER is claimed to provide mobility, true mobility,
with frequent MN changes of address, is to be provided by HIP or
Mobile IPv6, rather than by RANGER itself.  This precludes the
achievement of mobility for IPv4 hosts.

RANGER uses the SEAL tunneling and Path MTU Discovery
management.  Some limitations of this have been mentioned above
and are discussed in greater detail below.

Wide adoption of RANGER for scalable routing purposes would not
necessarily involve deeply recursive levels of enterprises.  A
single depth of "enterprise" would avoid the need for
concatenated tunnels, with decapsulation and re-encapsulation at
each depth of enterprise traversed.  Since RANGER's mapping
system returns different eEBR addresses depending on which
enterprise the mapping is looked up within, it is difficult to
envisage a simple mapping and tunneling arrangement comparable
to the unified systems used by other CES architectures.

draft-russert-rangers-01 page 15 has an example of how the
mapping systems in different nested enterprises produce
different results for the one EID address.  The packet is
tunneled within one enterprise to a router which decapsulates it
and looks up its the EID address' mapping in the mapping system
of the enclosing enterprise.  This causes the packet to be
tunneled to another router where the same process may occur for
the next level of enclosing enterprise.  Exactly how this would
work for ISP networks and end-user networks in a minimal
scalable routing scenario is difficult to understand.

Assuming an IPv4-only or IPv6-only implementation, for
simplicity, the global unicast address space would be the
commons of the Internet and a subset of this would be reserved
for the globally unique EID addresses.  The global unicast
addresses which are not included in any EID prefixes are then
known as Internet RLOC addresses.  If we assume that no network
re-uses any of this space, then "RLOC" denotes conventional
global unicast addresses which are unique within the whole
Internet and which are not subject to special attention by the
ITR-like ibex when found in the destination field of a packet.
However, private address space within each enterprise is widely
used, and is known as "RLOC" within RANGER, although it is in an
entirely separate namespace from global unicast address space.

An iEBR in any network needs to be able to look up mapping for
the EID prefix matching a destination address of a packet it
receives.  Assuming all iEBRs are preconfigured with information
about which prefixes constitute the EID subset of the global
unicast address space (which the IDs do not mention, but which
would be necessary), it is not clear how an iEBR proceeds to
find the mapping.

No specific mapping protocols are required by RANGER, so the IDs
are incomplete in this respect at least in terms of providing a
practical scalable routing architecture.  This is compounded by
the options of either requesting mapping or forwarding the
packet to an upstream "default mapper".

These mapping lookups appear to be distinct from the use of
redirects (which have no caching time) as a data-driven method
by which a router can discover the next-hop router for packets
matching a given prefix, because this approach only applies to
whatever is regarded as "RLOC" within the enterprise.

Section 3.1 provides the best guidance on mapping systems:

   The mapping system is typically maintained as a per-enterprise
   distributed database that is synchronized among a limited set of
   mapping agents.  Distributed database management alternatives include
   a routing protocol instance maintained by Enterprise Border Gateways
   (EBGs), a DNS reverse zone distributed among a restricted set of
   caching servers, etc.  Mapping entries are inserted into the mapping
   system through administrative configuration, automated prefix
   registrations, etc.

   RANGER allows for an ITE to either consult the mapping system itself
   (while delaying or dropping initial IP packets) or forward initial
   packets to an EBG acting as a "default mapper".  In either case, the
   ITE receives a mapping reply that it can use to populate its
   Forwarding Information Base (FIB).  The choice of mapping approaches
   must be considered with respect to the individual enterprise network
   scenario, e.g., forwarding to an EBG may be more appropriate in some
   scenarios while ITE self-discovery may be more appropriate in others.
   Use of other mapping mechanisms is also possible according to the
   specific enterprise scenario.

Also, section 3.7:

   Selection of mapping alternatives may have significant implications
   for applications, server selection, route determination, security,
   etc.  In particular, applications that expect all packets (including
   initial ones) to experience similar delays may be adversely affected
   by a scheme that imposes non-negligible delays when initial packets
   are queued while a look-aside mapping table is consulted.  Still
   other applications may experience significant startup delays when its
   initial packets are dropped during a mapping lookup event.  These
   factors would seem to favor a scheme that is able to forward initial
   packets along a path with sub-optimal delay while a mapping lookup is
   performed in parallel, e.g., such as when a "default mapper" is used.

   Generally speaking, proactive mapping-maintenance mechanisms may have
   scaling issues with the amount of updates they generate, while
   reactive mechanisms may involve effects to the delay of initial
   packets before the cached state is updated.  Also to be considered
   are attacks against the mapping mechanism, which may result in denial
   of service of the mapping cache.

The first RANGER ID is from October 2008 and the current version
is experimental RFC-5720-to-be.   Yet there is no defined
mapping system.  The choice of mapping system is widely regarded
as the biggest single difficulty in any CES architecture.

Nor is their any specification for, or guidance about:

  * Caching by iEBRs.

  * The technical and administratively authoritative sources of
    mapping.

  * How each enterprise's mapping system interworks with that of
    other enterprises.

  * Administrative arrangements for who controls which parts of
    the global unicast address space are used as EIDs.

  * Format of mapping requests.

  * Format of mapping replies and or how they can be secured when
    they result from either a mapping request or by the forwarding
    of an unencapsulated traffic packet to an upstream router which
    functions both as a default mapper and as an iEBR.

Although the system is supposed to support multihoming, there is
no mention of how an iEBR can learn of multiple eEBR addresses
to use, and how it could decide which of them to used based on
somehow discovering whether each one is currently able to
forward packets to the destination network.

Likewise, there is no discussion of mapping or iEBR mechanisms
for load-sharing or any other approach to inbound TE from the
point of view of the destination network.

There is no plan for gradual deployment, with full benefits to
early adopters, no matter how few have so far adopted RANGER in
their networks.  For instance, there is no apparent equivalent
of Ivip's DITRs or LISP's PTRs, both of which are essential to
correctly tunneling packets sent from networks which have not
yet been upgraded.

draft-russert-rangers-01 gives an example (Figure 12) of global
mobility with a passenger aircraft.  In this example, "the
worldwide Internet sees no change".  This is because all ground
stations are part of a single "Aero Enterprise" network with a
single point of connection to the Internet.  Consequently,
excessively long paths will frequently be encountered.  If the
network linked to the Internet in Paris and the aircraft was
using a ground station near Dunedin in the South Island of New
Zealand, then any packets sent by hosts in the aircraft to the
University of Otago there would need to travel from the ground
station, to Paris and then back, via the Internet, to Dunedin.
Likewise in the return path the path would be excessively long.

Please see:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

by Steve Russert (who co-wrote the ID in question) and myself
for an example of using the TTR Mobility architecture to avoid
such excessively long paths.

The RANGER IDs make several references to "Carrier Grade NAT".
Only one reference links to another document:

  An Incremental Carrier-Grade NAT (CGN) for IPv6 Transition
  http://tools.ietf.org/html/draft-jiang-incremental-cgn-00

This is somewhat different than "dual-stack-lite", but still has
the problems inherent in trying to server multiple customers,
such as home or SOHO networks, from a single IPv4 address.
These problems involve scaling difficulties in the CGN box and
the impossibility of more than one of the customers using port
forwarding for the same port.  This is needed for some p2p and
other applications.  The site

   http://www.portforward.com/cports.htm

lists 2221 applications which use port forwarding, most of them
apparently games, but also including various widely used
Instant Messenger applications.

RANGER aims to provide a general, powerful, recursive framework
for networking, based on the Catenet work of Louis Pouzin in 1974:

   By restoring the original CATENET vision to the Internet, the next
   generation Internet can be arbitrarily scalable while simultaneously
   supporting provider independence, mobility, multihoming and security.

However, as currently specified it is unsuitable for adoption as
a scalable routing architecture.

SEAL

An earlier comparison of SEAL and Ivip's IPTM protocol from
April 2008 is:

  SEAL & IPTM: differing goals
  http://www.ietf.org/mail-archive/web/rrg/current/msg02149.html

IPTM is described at:

  http://www.firstpr.com.au/ip/ivip/pmtud-frag/

Re-reading the latest version RFC-5320-to-be on 2010-01-26 I
concluded the following.  I am using "ITR" to denote RANGER's
encapsulating tunnel function (ETE or eEBR):

(4.2.1) SEAL does not attempt to deal with packets longer than
~1500 bytes.  A major goal of IPTM is to enable the ITRs to take
advantage of ~9k byte MTU paths to the ETRs.  These jumboframe
paths will, presumably, become common and later ubiquitous in
the DFZ, as well as within most ISP and end-user networks.
Therefore, adoption of seal would mean these jumboframe paths
would not be properly utilized.

This section, on page 11, states that the ITR could set the MTU
of the tunnel input to 1500 bytes - or somewhat more if it was
desired to cope with ~1500 bytes original packets which had
extra encapsulation by the time they reached the ITR.  The ITU
could also set the MTU as low as 1280 bytes for IPv6, and 576
bytes for IPv4, though it is not explained why such low values
might be considered.

With IPv4 DF=0 packets longer than this chosen MTU, the ITR
fragments them at that point before submitting the resulting
pair (or more than two?) of packets for SEAL processing.

With IPv6 packets and IPv4 DF=1 packets which exceed the chosen
MTU value, the ITR sends a PTB to the SH so future packet
lengths will not exceed the chosen MTU value.

Other than this, SEAL does not attempt to send RFC 1191 PTB
messages to the sending host.  Consequently, for maximal length
packets the same length as the chosen MTU, if this length plus
the SEAL and other encapsulation headers exceeds the actual PMTU
to the ETR, then the ITR and ETR will work together to adapt the
ITRs SEAL fragmentation of the resulting packet.

This means that if the chosen MTU, plus SEAL and other
encapsulation headers results in a figure above ~1500, then in
today's DFZ (which I understand typically has ~1500 byte PMTUs)
each maximal length packet the SH generates will result in the
ITR emitting two separate packets.

To avoid this unsustainable performance and reliability
degradation, SEAL would need to be operated with a low-enough
chosen MTU to ensure the SH's packets are never long enough to
exceed DFZ PMTUs once the SEAL and other encapsulation headers
have been added.   This would be an MTU value substantially
below 1500 bytes, and would be marginally shorter for IPv6
because of its longer IP header.

In summary:

SEAL would prevent the use of jumboframes in the DFZ for all
tunneled packets.  (Until perhaps some time in the future where
all ITRs could reach all ETRs across the DFZ, and via all other
routers, with jumboframe PMTUs - then all SEAL tunnel input MTUs
could be set to some value a little below ~9kbytes.)

Even if a greater than 1500 byte PMTU was available between the
ITR and the ETR, SEAL would not allow it to be used fully, since
the MTU of the tunnel input would (in order to avoid
fragmentation of all maximum length packets) be set to a value
substantially below 1500 bytes.

SEAL has quite a complex protocol for detecting PMTU to the ETR
compared to IPTM.

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Jan 2010 13:51:23 -0800

Robin,

I will respond to the RANGER portion of the critique later,
but in terms of the SEAL portion please be sure you are
evaluating the correct version. The latest SEAL spec is
found here:

  http://tools.ietf.org/html/draft-templin-intarea-seal

and fully supports jumbograms.

Thanks - Fred
[email-masked]

> -----Original Message-----
> From: Robin Whittle [mailto:[email-masked]]
> Sent: Tuesday, January 26, 2010 3:48 AM
> To: RRG
> Cc: Templin, Fred L
> Subject: RANGER and SEAL critique
>

---

From: Robin Whittle <[email-masked]>
Date: Wed, 27 Jan 2010 11:53:18 +1100

Hi Fred,

I am keen for you to provide a guide to exactly how RANGER would
be used for scalable routing, with details of the mapping system,
multihoming, TE and mobility etc. and why this would be a better
choice than LISP or Ivip.

The only other CES architecture, TIDR, can't solve the biggest
single scaling problem - the burden on the DFZ control plane due
to each additional end-user prefix.

The 6 or so CEE proposals are not suitable for widespread
voluntary adoption, and I think I will argue against them on
other grounds as well.  The remaining proposals are not full
solutions to the routing scaling problem.

So that leaves LISP, RANGER and Ivip.

> and fully supports jumbograms.

Yes - I was referring to an earlier version.

My critique of RANGER was based on:

   http://tools.ietf.org/rfcmarkup?rfc-repository=http://www.rfc-editor.org/authors&doc=rfc5720&topmenu=true&document=draft-templin-ranger-09&docreplaces=draft-templin-ranger-09&title=RFC-EDITOR+AUTH48+REVIEW+COPY&extrastyle=body+{background-color:%23fee%3b}

This is an RFC-to-be 5720 "January 2010" which I found by
clicking a link "RFC-to-be" at the top of:

   http://tools.ietf.org/html/draft-templin-ranger-09  (26 October 2009)

My critique of SEAL was based on:

   http://tools.ietf.org/rfcmarkup?rfc-repository=http://www.rfc-editor.org/authors&doc=rfc5320&topmenu=true&document=draft-templin-seal-23&docreplaces=draft-templin-seal-23&title=RFC-EDITOR+AUTH48+REVIEW+COPY&extrastyle=body+{background-color:%23fee%3b}

This is an RFC-to-be 5320 "January 2010" which I found by
clicking a link "RFC-to-be" at the top of:

   http://tools.ietf.org/html/draft-templin-seal-23  (28 October 2008)

The RANGER summary:

   http://www.ietf.org/mail-archive/web/rrg/current/msg05665.html

links to draft-templin-ranger-09, so it seems I was looking at
the correct RANGER document.  I see now that the summary links to
the later SEAL document "draft-templin-intarea-seal-08".

Searching for SEAL with the Google option at:
http://tools.ietf.org/html/ will lead most people to the
"draft-templin-seal-23" page, since this is currently the first
search result.

Can you terminate this zombie strand of SEAL IDs, especially
since it leads to an apparently up-to-date RFC-to-be?

This prompted me to realise I have a dead-end ID:

  draft-whittle-ivip4-etr-addr-forw-01

which is continued under another name, without the '4'. I will
post a short -02 version to point people to the new IDs.

In http://tools.ietf.org/html/draft-templin-ranger-09 the SEAL
reference does link to draft-templin-intarea-seal-07, but in the
RFC-to-be - which is a later version of draft-templin-ranger-09 -
the SEAL reference doesn't link to anything.  It does give a
later date - October 2009 - than the October 2008 date of
draft-templin-seal-23, but that is easily missed.  So people
are likely to use the Google facility and pick the top result,
which has an impressively high version number, and then
follow the link to the RFC-to-be 5320, which displays a January
2010 date.

I will look at the latest SEAL draft:

  http://tools.ietf.org/html/draft-templin-intarea-seal-re-02

 - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 26 Jan 2010 17:00:45 -0800

Robin,

I will have something to say about RANGER soon (can't
do it right now) but just so there is no confusion the
correct url for the latest SEAL document is this one:

Thanks - Fred

---

From: Robin Whittle <[email-masked]>
Date: Wed, 27 Jan 2010 15:56:35 +1100

Short version:    "Jumbograms" are not the same as "jumboframes".

                  Discussion of SEAL's support for packets longer
                  than 1500 bytes and up to 64k bytes - these are
                  "jumboframes".

                  Questions about how SEAL detects the PMTU being
                  too short for the currently sent packets - it
                  appears to be based on detecting fragmented packets
                  for IPv4, and by using ICMP PTBs for IPv6.  If so
                  why not use ICMP PTBs for IPv4 too?

                  When would SEAL segmentation be used?  I would
                  have thought it is never desirable to have
                  ITRs, ITEs or any intermediate router fragmenting /
                  segmenting packets into two or more smaller packets
                  on an ongoing basis.  Surely the sending host
                  should be sent a PTB to cause it to send packets
                  which, once encapsulated, will fit within the
                  actual PMTU of the ITR/ITE path to the ETR/ETE.

Hi Fred,

I read the current ID:	

  http://tools.ietf.org/html/draft-templin-intarea-seal-08

I used the term "jumboframe" to denote IPv4 or IPv6 packets which are
 up to about 9kbytes long, as opposed to the conventional length of
~1500 bytes.

Your current ID and your previous mailing list message used the term
"jumbogram", which is IPv6 only and refers to packets of 65,575 bytes
or more (up to 4 gigabytes) - and which have a "Jumbo Payload" option
header.

My IPTM approach needs to be written up more clearly.  As far as I
know, it is the only thing which attempts anything similar to SEAL -
although it is purely for use within Ivip, and assumes that the outer
source address of ordinarily tunneled packets is that of the sending
host, not the ITR.

IPTM doesn't have any particular upper limit for packet length, other
than that imposed by the 16 bit length fields in the IPv4 and IPv6
headers.   Any such packet longer than 1500 bytes is properly
referred to as a "jumboframe".  IPTM is not intended to support
"jumbograms".

I think your SEAL proposal is supposed to work with the same - IPv4
and IPv6 packets with normal headers and lengths up to 65,575 bytes.
 If so, then I think "jumbogram" is not the correct term to describe
SEAL's capabilities.

Assuming that you do aim to handle both IPv4 and IPv6 packets up to
65,575 bytes, then here is my understanding of how SEAL-SR does it.

There is new text on page 15: "The ITE can alternatively ...".
Before this text is guidance on how the ITE can set an MTU value on
the input of the tunnel to 1500 bytes or greater, or significantly
lower, such as to 576 bytes for for IPv4 and 1280 bytes for IPv6.  It
is not very clear how such decisions should be made.  The new text
provides a third option:

   The ITE can alternatively set an indefinite MTU on the tunnel
   virtual interface such that all inner IP packets are admitted into
   the interface without regard to size.  For ITEs that host
   applications, this option must be carefully coordinated with
   protocol stack upper layers, since some upper layer protocols
   (e.g., TCP) derive their packet sizing parameters from the MTU of
   the underlying interface and as such may select too large an
   initial size.  This is not a problem for upper layers that use
   conservative initial estimates, e.g., when mechanisms such as
   Packetization Layer Path MTU Discovery [RFC4821] are used.

I understand this as meaning that the ITE (AKA ITR in other CES
architectures) will accept any size packet initially, but may later
change the limit to a lower value and send PTBs to the sending host
(assuming the SH is sending IPv6 packets or IPv4 packets with DF=1).

As far as I know, no applications use RFC 4821.  Can you cite any
which do?  RFC 4821 involves different applications sharing
information via the operating system about their experience with
packet lengths sent to a particular destination host.  This sounds
complex to implement and text - and I guess there is little
motivation for early adoptors amongst application writers if few
other apps do it.

Also, the "ITE which hosts applications" is confusing, since in
RANGER, the ITE is a router.  Maybe you mean apps inside the router
which use TCP.

My overall understanding is that if the ITE adopts this indefinite
MTU, and adjusts it downwards if any packets are too big for the PMTU
to the ETR, then you can cope with IPv4 and IPv6 packets up to 65,575
bytes.  If so, then this paragraph of my critique no longer applies:

  SEAL.  ...

However I think the ID should be amended to avoid reference to
"jumbograms".

As best I understand it, SEAL sends all its encapsulated traffic
packets from the ITE to ETE (ITR to ETR) with:

   IPv4  DF=0

   IPv6  with a Fragmentation header

I understand the purpose with IPv4 DF=0 is so if the packets are too
long for the PMTU to the ETE, that they will be fragmented and the
ETE will therefore receive at least some of the fragments, and so
report this to the ITE (4.4.5.1.2).  This is how SEAL works to detect
PMTU problems in the tunnel - rather than relying on RFC 1191 ICMP
PTB packets generated by routers between the ITE an ETE, at least for
IPv4.

In IPTM (for both IPv4 and IPv6) I send a pair of packets, one short
and one long, (perhaps a few copies of the short one, to improve
robustness) for any traffic packet which is longer than the ITR has
previously successfully sent to the ETR.  The ETR responds to the
shorter one, and tells the ITR if the long one didn't arrive.  (Also,
the ETR will reply to the ITR if only the long one arrives.)  These
are DF=1 packets for IPv4 and ordinary (implicitly DF=1) packets for
IPv6.

I think it is bad to send DF=0 packets into the DFZ and expect the
routers to dutifully fragment them if they are too long - though I
know this should not happen frequently with SEAL, since it is just
part of detecting the PMTU.

Do DFZ routers handle IPv6 packets with the Fragmentation Header just
as fast as those without?

With IPv6, does SEAL detect its encapsulated packets being too long
for the ITE to ETE PMTU by relying on ICMP PTB messages from the
limiting router in the ITE to ETE path?  (AFAIK, IPv6 packets are
never fragmented by ordinary routers.)  If so, then why not do the
same with IPv4 packets?

Assuming the above-mentioned approaches enable each SEAL ITE to
discover the actual PMTU to a given ETE (and to probe occasionally to
see if the value has increased), and assuming it uses this to send
PTB packets to the SH in order that subsequently emitted packets will
be of the right size, once encapsulated, to not exceed this PMTU,
then what is the purpose of the SEAL segmentation system?

   Actually, it is trickier.  The one SH may be sending to multiple
   destination hosts (DHs) all of which are for one or more EID
   prefixes for which the ITE is currently tunneling to a given ETE.
   Then the ITE needs to send the SH a separate PTB for each too-long
   packet it sends to each such DH.  Also, there may be multiple such
   SHs using this ITE whose packets will be tunneled to this ETE.

If the SH sends a really long IPv4 DF=0 packet, the ITE will use
ordinary IPv4 fragmentation to turn it into multiple shorter fragment
packets, before SEAL processing.  (I think, come the Revolution, that
DF=0 packets longer than some value like 1450 bytes or similar should
be *dropped* - applications are expecting too much of the Network to
fragment and carry the fragments of their excessively long packets.
Applications have had since 1991 - RFC 1191 - to change their
free-loading ways.)

Under what circumstances would SEAL segmentation be used?

IPTM does not do any such thing, except for the rare dual-packet
protocol which is used for probing PMTUD to the ETR.

I would have thought it highly undesirable to allow SHs to keep
sending packets which the ITE, ITR or whatever should have to turn
into two or more packets just to get to the ETR.

  - Robin

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 27 Jan 2010 07:50:02 -0800

Hi Robin,

Thanks for taking another look at SEAL. Please see below
for just a few comments in response:

Fred

> Sent: Tuesday, January 26, 2010 8:57 PM
> To: RRG
> Cc: Templin, Fred L
> Subject: Re: [rrg] RANGER and SEAL critique
> 

The terminology is has become a bit loose here since
the GigE community began using the term "jumbo" to
refer to their 9KB target. For SEAL, what I am
meaning to say is that SEAL-SR can handle all sizes
from 1500 Ethernet size (or smaller) on the low end
up to all sizes of IPv6 jumbograms on the high end.
I'll see if I can disambiguate in the text. 

> motivation for early adoptors amongst application writers if few

Applications do not know whether they are running on
a stack that uses RFC4821 or just classical PMTUD;
the use or non-use of RFC4821 is a stack consideration
and not an application consideration.
 
> RANGER, the ITE is a router.  Maybe you mean apps inside the router

Yes; app's inside the router which use TCP. I guess
we can use MSS clamping for apps inside the router
if we are concerned about them getting too large of
an initial PMTU estimate.

> are DF=1 packets for IPv4 and ordinary (implicitly DF=1) packets for

I can see how the ETR can send a response for packets that
arrive, but then how long must the ITR wait to receive the
response? And, what if the response is lost?

I can't see how the ETR can send a response for packets
that *don't* arrive. For example, what if the large
packet was dropped due to congestion and not due to
an MTU limitation? And, for how long must the ETR wait
before sending the NACK?

IMHO, DF=1 for PMTUD is busted, and these are a few of
the reasons why.

> part of detecting the PMTU.

In-the-network fragmentation is a noise indication, and
SEAL moves quickly to tune out the noise.
 
> as fast as those without?

I don't honestly know, but I sure hope that they do.
Anyone from the major router vendor community care
to comment on this?
 
> never fragmented by ordinary routers.)  If so, then why not do the

ICMPv6's are guaranteed to include a sufficient portion
of the original packet whereas ICMPv4's are not. Hence,
the theory is that the ITE can "match up" the ICMPv6s
with actual packets that cause the PTB. Also the ICMPv6's
can readily be translated into PTBs to send back to the
original source while ICMPv4's may not contain enough
data for translation.

Other than that, both ICMPv6's and ICMPv4's can be dropped
by filtering gateways so both IPv6 and IPv4 are susceptible
to black-holing. But, for the non-fragmenting IPv6 network
using DF=1 this is the best we can do; for IPv4 we can do
better by using DF=0.    
 
> Under what circumstances would SEAL segmentation be used?

SEAL is intended not only for the core routers we usually
talk about in RRG, but also for any ITR/ETR routers
deployed in any operational scenario. For example, we
would never ask core routers to do segmentation and
reassembly (hence the recommendation for SEAL-FS in
those environments). But, we might want to ask routers
in edge networks (e.g., MANET routers, CPE routers in
ISP networks, etc.) to do segmentation and reassembly
in order to avoid having to constantly tell hosts to
reduce their PMTU estimate to a degenerate size (i.e.,
anything less that 1500). So, in those environments
we would recommend SEAL-SR.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 27 Jan 2010 17:34:05 -0800

Robin,

Here is what I think about RANGER and scalable routing.
RANGER expects that the existing state of affairs in the
current Internet BGP routing system will persist, but the
goal of RANGER is to arrest the growth of the BGP RIB so
that it will level off and not continue to expand along
super-linear rates. In particular, RANGER expects that
the current BGP will continue to maintain the RLOC-based
RIB for the Internet, but that future growth due to
mobility, multihoming and PI addressing will be handled
out of EID space instead of RLOC space.

RANGER asks that a new BGP instance that carries EID
prefixes be established within the DFZ, where each
participating EID-based BGP router is an ITR/ETR that
treats the DFZ as a virtual NBMA link through tunneling.
Each participating EID-BGP router will set up peering
arrangements with a limited set of neighbors using
tunnels according to the NBMA link model. There is no
requirement that these EID-BGP routers also participate
in the current RLOC-based BGP routing instance, so the
EID-BGP routers can be deployed incrementally and
without disturbing the existing RLOC-BGP routing system.

This new EID-BGP instance will be used for carrying a
relatively small number of highly-aggregated EID
prefixes in keeping with the principles of Virtual
Aggregation (VA). So, this new EID-BGP instance (which
again is completely separate from the existing RLOC-BGP
instance) will carry only highly-aggregated Virtual
Prefixes (VPs) such as 4000::/8, 4100::/8, etc. So, at
most there will be perhaps a few thousand of these VPs
in the EID-BGP RIB (or perhaps even a few 10's or 100's
of thousands) but the RIB size will be kept manageable
through VA.

Now, RANGER has the EID-based VPs populated throughout
the EID-BGP RIB, with all of the EID-BGP routers
connecting service provider (SP) networks via the virtual
NBMA link configured over the core. Customer Edge (CE)
routers within the SP networks will want to use EID-based
PI prefixes. Each such CE router "registers" its EID PI
prefixes both within the SP network and with the EID-BGP
routers that own the VP from which the PI prefix is
aggregated. Once "registered", the CE's PI prefix will
be kept only in selected router FIBs, and will not be
injected into the EID-BGP RIB. Moreover, only the FIBs
of those routers on the paths over which the CE's EID
addressed packets will travel need to contain the PI
prefix - no other routers need discover the prefix. The
location of the CE router's EID prefix is tracked through
the FIB entries in the EID-BGP router that holds the VP
from which the EID prefix is derived.

In summary, the RANGER approach to scalable routing is
to create a new BGP instance between tunnel routers for
the purpose of keeping a limited set of highly-aggregated
EID VPs in the RIB. PI EID prefixes owned by customer
routers are added to selected SP router FIB tables on
demand, and are never injected into the RIB. The way
this works is that CE routers that are holders of PI EID
prefixes "blow bubbles" that percolate up through a reverse
tree ascending through their SP networks until the bubbles
reach an EID-BGP router that owns a VP from which the PI
prefix is derived. In that way, the locations of all PI
EID prefix holders are available in EID-BGP router FIBs
while only VPs appear in the EID-BGP RIB. This system
of knowing where all PI prefix holders are at all times
also has clear beneficial properties for supporting
mobility and multihoming.

Finally, in terms of routing scaling, the end state
benefit is that both the EID-BGP and RLOC-BGP RIBs
remain manageable in size and only those routers that
need to know about certain PI EID prefixes have to
carry those prefixes in their FIBs.

Any thoughts or comments on this?

Fred

> Sent: Tuesday, January 26, 2010 4:53 PM
> To: RRG
> Cc: Templin, Fred L
>    http://tools.ietf.org/rfcmarkup?rfc-repository=http://www.rfc-
> editor.org/authors&doc=rfc5720&topmenu=true&document=draft-templin-ranger-09&docreplaces=draft-
> templin-ranger-09&title=RFC-EDITOR+AUTH48+REVIEW+COPY&extrastyle=body+{background-color:%23fee%3b}
> 
>    http://tools.ietf.org/rfcmarkup?rfc-repository=http://www.rfc-
> editor.org/authors&doc=rfc5320&topmenu=true&document=draft-templin-seal-23&docreplaces=draft-templin-
> seal-23&title=RFC-EDITOR+AUTH48+REVIEW+COPY&extrastyle=body+{background-color:%23fee%3b}
>

---

From: Robin Whittle <[email-masked]>
Date: Thu, 28 Jan 2010 14:26:00 +1100

Short version:   Can any router folks confirm whether or not IPv6 DFZ
                 routers handle packets with the Fragmentation Header
                 just as quickly and efficiently as ordinary packets?

                 I argue against Fred Templin's position that
                 ordinary RFC1191 DF=1 Path MTU Discovery (and
                 therefore its RFC1981 IPv6 equivalent) is "busted".

                 Where is the evidence that networks filtering out
                 PTB (Packet Too Big) messages is a significant
                 problem?

                 To the extent that any such problem exists, why
                 should this be accepted and further protocols
                 created to work around it?  I think the networks
                 which do this are non-compliant with the inter-
                 working requirements of the Internet.  These
                 networks should change their ways.

                 I argue that RFC4821 Packetization Layer PMTUD is
                 unnecessary, over-complex and therefore undesirable.
                 Furthermore I assert it is *vapourware* - since I
                 can't see any evidence of applications or stacks
                 which actually use it on user traffic packets.
                 (There is a minimal implementation in the Linux
                  stack, which is turned off by default.)

                 My IPTM PMTUD system could work fine with minimal
                 length IPv4 RFC1191 PTB messages.

                 Fred says that his SEAL system couldn't work with
                 minimal IPv4 PTBs, but I suggest that if the
                 SEAL header was extended from its current 32 bits
                 to 64, with a 32 bit nonce, that it would work fine.
                 This would avoid the need for his current approach
                 of sending DF=0 packets, and having them fragmented
                 as a method of detecting an MTU problem.

Hi Fred,

Thanks for your reply in the "Re: [rrg] RANGER and SEAL critique"

> I'll see if I can disambiguate in the text. 

I had never made the distinction between "jumboframe" and "jumbogram"
- but I only used the former.

When reading your SEAL ID, I looked them up:

  http://en.wikipedia.org/wiki/Jumbogram

This points to RFC 2675 - "IPv6 Jumbograms":

  http://tools.ietf.org/html/rfc2675

which explicitly refers to packets with a special "Jumbo Payload
option" - with lengths up to 4 gigabytes.

  http://en.wikipedia.org/wiki/Jumbo_frame

Jumboframes are ordinary format packets up to 64kbytes - and this is
what SEAL works with.

> and not an application consideration.

That's not my recollection . . .

There are potentially multiple packetization layer systems in a
single host.  One is the TCP layer in the stack.  Others could be in
applications.  Perhaps there would be an SCTP layer in the stack too.

All these are supposed to share information - but there's no
standardized way of doing this.

   Most of the difficulty in implementing PLPMTUD arises because it
   needs to be implemented in several different places within a
   single node.  In general, each Packetization Protocol needs to
   have its own implementation of PLPMTUD.  Furthermore, the natural
   mechanism to share Path MTU information between concurrent or
   subsequent connections is a path information cache in the IP
   layer.  The various Packetization Protocols need to have the means
   to access and update the shared cache in the IP layer.  This memo
   describes PLPMTUD in terms of its primary subsystems without fully
   describing how they are assembled into a complete implementation.

So theoretically a TCP layer of a stack could do PMTUD not only with
the conventional RFC1191 (IPv4) and RFC1981 (IPv6) approach, but
could also attempt to discern the PMTU to a destination host based on
the assumption that ICMP PTBs were not being received, and that the
need to repeatedly retransmit packets above a certain size was good
enough evidence of a PMTU limit to decide the PMTU was a lower figure
than it would otherwise have defaulted to.

The only deployed implementations of RFC4821 seems to be a minimal
one, turned off by default, in the Linux kernel.  I guess this is
just for TCP.  Does it have any way of sharing information with other
packetization layers, in the IP stack or in applications?

If an application had its own packetization code, for instance to
decide how big to make UDP packets, then it would need to be able to
get from the stack any information which might have been placed there
(a Path MTU figure to this destination host) by the TCP RFC4821
layer, when previously sending packets to this destination host - or
by other applications' packetization layers.  Also, there would be a
need to update this information - and at the same time inform all
other RFC4821-aware packetization layers of the changed MTU value:

   If PLPMTUD updates the MTU for a particular path, all
   Packetization Layer sessions that share the path representation
   (as described in Section 5.2) SHOULD be notified to make use of
   the new MTU and make the required congestion control adjustments.

Applications with their own packetization layers are definitely involved:

9. Application Probing

   All implementations MUST include a mechanism where applications
   using connectionless protocols can send their own probes.

   . . .

   Implementing PLPMTUD in an application also requires a mechanism
   where the application can inform the operating system about the
   outcome of the probe as described in Section 7.6, or directly
   update search_low, search_high, and eff_pmtu, described in Section
   7.1.

So I disagree with your statement except to the extent of
applications which do no packetization and just rely on the stack's
TCP (or SCTP or whatever) packetization layers which may or may not
implement RFC4821.

Matt Mathis, who co-wrote RFC4821 has a page devoted to MTU matters:

  https://www.psc.edu/~mathis/MTU/

There's no mention of SEAL or my IPTM approach.  There's a mailing
list which I just joined.  There have been 16 messages since 2005.
The archives are not public.  I will post some links to this SEAL
discussion.

There were only about a hundred messages in the IETF PMTUD mailing
list over the 8 months of its activity which resulted in the
standards track RFC4821:

  http://www.ietf.org/mail-archive/web/pmtud/current/maillist.html

My impression is that this is pretty minimal for a standards track
RFC concerning a problem affecting potentially all applications.

The state of RFC4821 adoption is:

  https://www.psc.edu/~mathis/MTU/#pmtud

     Implementations:

       * Linux 2.6.17: ICMP black hole recovery only, off by default

     As we gain field experience with wide deployment of RFC 4821 in
     the above two configurations, we will document any additional
     recommendations for implementors. Watch this page for future
     information.

I interpret this to mean that the single potentially used
implementation is minimal and is turned off by default.  I guess it
is just for TCP.

I always suspected that RFC4821 was never going to be widely adopted.

Firstly I can't find evidence of widespread problems with RFC1191 /
RFC1981 PMTUD.  I guess there are glitches, but if any network is
dropping ICMP PTB packets, then IMHO the administrators need to
change their ways.  This is a simple question of network
administration being at odds with the needs of a perfectly good
protocol, which normally works fine all over the Net.

The solution to bad network administration is not a new protocol.

Where is the evidence of widespread PMTUD problems?

Secondly, RFC4821 is extremely hard to implement.  It requires
communication back and forth between packetization layers in multiple
applications plus TCP and maybe SCTP (others in the future?) in the
stack.  Why should these trust each other?  How can this sort of
thing be debugged in actual operation in sending hosts around the
world?  It sounds like a development nightmare.

It seems it has not been implemented beyond a subset being put in the
Linux kernel and turned off by default.

Here are my current beliefs:

  1 - There is no widespread problem with RFC1191 PMTUD (or its IPv6
      equivalent RFC1981).

  2 - RFC1191/1981 is both necessary and sufficient to solve PMTUD
      problems.  It will work fine as long as the limiting router
      sends a PTB message and as long as nothing drops these
      messages by filtering.  (However, perhaps the fragment of
      packet sent back in the IPv4 PTB message should be made
      somewhat longer so that more information - such as a nonce in
      the payload behind encapsulation headers - can be found.  Its
      my impression that quite a few routers already do this.
      The ICMPv6 PTB specifies "As much of invoking packet as will
      fit without the ICMPv6 packet exceeding 576 octets.")

  3 - Therefore, any problems with RFC1191/1981 PMTUD are caused by
      network maladministration, and should be solved there.

  4 - So there's no genuine need for RFC4821.

  5 - Any document which points to RFC4821 as if it is, or will
      be, adopted to any significant degree is misleading.

I think your SEAL documents do this - point to RFC4821 as if it is a
legitimate, desirable or even superior method of doing PMTUD.  I
think RFC4821 is unnecessary and undesirable - and that it is vapourware.

SEAL is designed not to depend on RFC1191 PTB packets - but you do
rely on RFC1981 packets.  More on this below.  You go to a lot of
trouble to achieve end-to-end confirmation of packets arriving intact
or being fragmented en-route (at last for IPv4).  But if the Net is
going to be upgraded with a bold initiative based on RANGER, surely
part of this upgrade can be to insist that no-one filter out PTB
packets.  Also, if you can implement something as far-reaching as
RANGER, then you can also devise updated versions of RFC1191 to
return longer segments of packets, to include wherever a nonce is
likely to be, even with packets with considerable levels of extra
encapsulation.  For IPv6, the router sends enough of the packet to
achieve this already.

> an initial PMTU estimate.

OK.

. . .

> response? And, what if the response is lost?

I haven't completed the design - I expect the ITR should retry after
a second or so.  If it can't get any response after a few retries,
then it should conclude, for the next few minutes or so, that this
ETR is not reachable.  Normally the ITR does no reachability testing
for ETRs.  But in this case, it discovers non-reachability as a
by-product of PMTUD management.  It should probably not send any long
packets to this ETR address for a while - and then only with the
short ones as part of another attempt to probe the PMTU to it.

> an MTU limitation? And, for how long must the ETR wait

Again, I need to work on the details.  I think that if the ETR gets a
short packet and no long packet arrives within 0.2 or 0.4 seconds or
so, then the long packet is never going to arrive.  So it should send
its response then.  ETRs are supposed to be connected via fibre, DSL
or some other wired arrangement. It would be wrong to put an ETR on
some long latency link.  Likewise, it would be a bad idea to put an
ITR on a long latency link.  (TTR Mobility complies with these
constraints.)

> IMHO, DF=1 for PMTUD is busted, and these are a few of

That's a very big statement which goes beyond your critique of how I
might implement IPTM.

Why do you believe DF=1 for PMTUD is busted?

Every application today uses it.  None of them use RFC4821.

Are all applications today breaking due to RFC1191 PMTUD being busted?

I see no evidence of this whatsoever.

> SEAL moves quickly to tune out the noise.

OK - I just think that DF=0 packets should be avoided in general.

DF=0 is an indulgence still granted to lazy bourgeois hosts in their
dotage, enabling them to continue their regressive habits of unfairly
burdening the proletariat of Worker routers with the responsibility
for slicing up and individually carrying and reassembling their
overweight, carelessly emitted, packets.  Come the Revolution . . . .

When IPTM sends a big packet (containing most of a traffic packet) as
part of its PMTU probing, if this hits an MTU limit, then it is
dropped, with a short PTB going back to the ITR.  With SEAL, in IPv4
mode, the limiting router has to split the packet up and forward it,
so other routers and the ETR have to carry all the fragments.  So my
IPTM approach is arguably lighter weight than your SEAL approach.

IPTM doesn't rely on the PTB. (See below for how it will be able to
work with minimal length IPv4 PTBs.)  As long as a PTB does get to
the ITR - which it would in most cases - then the ITR knows about the
MTU problem without having to wait for the ETR to time out and send a
message to the ITR saying the big packet did not arrive.  Also, the
ITR gets an exact MTU value from this PTB, rather than having to do
what SEAL does - hunt back and forth to find a packet size which is
reliably delivered without MTU problems.

> Anyone from the major router vendor community care

I am not suggesting they don't - but it would be good to find out.

> original source while ICMPv4's may not contain enough

OK - thanks for explaining the reasons for these different approaches.

I think these deficiencies with IPv4 PTBs relate to any routers which
only return the minimum number of bytes specified in RFC1191, which
is the IPv4 header and the next 8 bytes.

I recall people writing in the past that most routers send back more
than this.

I understand that all TCP layers in all IPv4 stacks rely on RFC1191
PMTUD for their proper operation.  As far as I know, there are no
problems with this, other than whatever level of difficulty is caused
by the occasional network dropping PTB packets.  However, this is for
packets which are being forwarded in their original form.

I guess applications and encryption functions within them deal
achieve IPv4 PMTUD via the same RFC1191 mechanisms.

It is trickier with an ITR using encapsulation to tunnel a traffic
packet to an ETR.

When an ITR encapsulates the packet, and the limiting router only
sends back 8 bytes after the outer IPv4 header, then there is not
enough of the inner packet to send back a valid PTB to the sending
host.  That's OK if the ITR caches enough of the original packet for
this purpose.  The real question is whether the returned IPv4 header
and the next 8 bytes enables the ITR to:

  1 - Match the PTB against a packet it tunneled to this ITR, so
      that the ITR knows which sending host the original packet
      came from - and to identify the correct initial part of that
      packet in its cache (which exists for this purpose alone).

  2 - Whether the ITR can recognise this securely enough to avoid
      spoofing attacks.

In Ivip, most traffic packets are encapsulated by the ITR with the
sending host's address as the outer header's source address.  Any PTB
which results from those goes to the sending host, which will not
recognise it.

Ivip's PMTUD management is done via the ITR using the IPTM (also my
invention - yet to be fully designed) protocol to carry traffic
packets in a pair of probe packets, whenever the traffic packet, if
normally encapsulated, would exceed the lower limit of the ITR's Zone
of Uncertainty about the real MTU to this particular ETR.

Both these packets are UDP packets.  The long one (B) has the ITR's
address in the outer header.  The smaller one (A) - of which multiple
may be sent for inexpensive robustness improvement - uses the sending
host's address in the outer header.

Here I consider how IPTM:

would work if the router only sent back 8 bytes following the IP
header of the long probe packet.  (This is only for these long probe
packets, which are accompanied by a shorter packet, or multiple
copies of the shorter packet - but the shorter packets will never
result in valid PTBs.)

If IPTM accepted a PTB from a long (B) packet, purely on the basis of
the IP header and the next 8 bytes being returned, and assuming this
set of probe packets resulted from a traffic packet specifically
generated by the attacker, then what would the attacker need to guess
in order to successfully have the ITR accept the spoofed PTB packet.

The aim of the attacker is DoS by having the ITR accept a lower than
genuine PMTU to this ETR - so it is not a disaster, just a cause of
less efficiency for the next ten minutes or so before the one or more
affected sending hosts tries its luck with a larger packet, as is
allowed and expected by RFC1191.

In this scenario, the ITR gets back just the IPv4 header and the UDP
header.  The attacker has to guess the 16 bit ID field in the IPv4
header, which is tricky - but it could eventually succeed in doing
so.  Here are the components of the UDP header:

  Source port     The ITR could use a randomized source port.  This,
                  combined with the 16 bit ID field, could extend
                  the number of bits to be guessed to 32 - which
                  I think is sufficiently secure, considering a
                  successful attack only degrades efficiency, rather
                  than causes actual loss of connectivity.

  Destination port   Currently, I assume there is a single UDP port
                  on all ETRs to send the long (B) packet to.  If
                  I could easily randomize this too - such as making
                  the most significant 8 bits fixed, and the others
                  up to the ITR to choose.

                  This would be 40 random bits - perfectly secure
                  considering the moderate level of DoS the attack
                  could result in.

  Length          If the attacker created the traffic packet, they
                  would know the length of what follows the UDP
                  header.

  Checksum        Ahhh - this is not a header checksum.  This covers
                  the data behind the UDP header.  This data is
                  mainly from the traffic packet, but it contains
                  a nonce.  So the 16 bit checksum is affected by
                  the nonce.

I hadn't realised this before - the UDP checksum contains another
16 bits the attacker has to guess.  Combined with the IPv4 header's
16 bit ID field, I think this makes it highly secure.  If this is
not enough, the 16 bit random ITR source UDP port should be sufficient.

So the ITR doesn't need any more bits than are necessarily supplied
by a minimally compliant RFC1191 implementation in the router which
sends the PTB.

How would this work for SEAL?

At present, for IPv4 and IPv4, your ITE (ITR) functions emit packets
with an outer header of IPv4 or IPv6, followed by a 32 bit SEAL header.

Immediately following the SEAL header you may have some "mid-layer
headers" which I don't properly understand.  Then you have the IPv4
or IPv6 traffic packet, or perhaps a segment of it.

You could make the SEAL ITE work fine with minimal length IPv4 PTBs
if the SEAL header was extended to 64 bits, with the additional 32
bits being a nonce.  That would always be returned in any PTB.

So I think your objection to using RFC1191 PTBs should only be based
on your concern about the PTBs being systematically dropped due to
filtering.

I assert that such filtering is a symptom of a badly administered
network - and that it should be fixed in the network, not worked
around with a protocol such as SEAL or IPTM.

> using DF=1 this is the best we can do; for IPv4 we can do

I just showed you can do both IPv4 and IPv6 the same way with a nonce
in the SEAL header.

What evidence can you show me for filtering of PTBs being a
significant problem in today's Internet?

To the extent that this is true, surely if you can upgrade the Net
along the lines of RANGER, then you can have a BCP that any network
which filters PTBs is not compliant with the basic requirements of
the Net.

> those environments). 

OK.

> But, we might want to ask routers

> Thanks - Fred

OK - this makes sense.  Thanks for discussing this in detail.

 - Robin

---

From: Robin Whittle <[email-masked]>
Date: Thu, 28 Jan 2010 14:28:45 +1100

Hi Fred,

Thanks for your speedy response to my RANGER critique.

I plan to reply in the next day or two, because I am trying to
concentrate on Name Based Sockets.

  - Robin

---

From: Tony Li <[email-masked]>
Date: Wed, 27 Jan 2010 23:25:07 -0800

>                  PTB (Packet Too Big) messages is a significant

This happens.  Consult some operator folks, privately and quietly.  Many 
enterprises blocked all inbound ICMP when DDoS attacks started happening.

>                  networks should change their ways.

You have the cart before the horse.  Reality is not warped to fit the 
needs of the network architecture, the network architecture must be 
molded to deal with reality.  The fact of the matter is that today, we 
have a hostile environment.  Anything and everything that can be used to 
create an attack will be used.  To protect themselves, many people will 
happily throw the baby (or at least its signaling protocol) out with the 
bathwater.

As a result, ICMP is dead.  Long live ICMP.

Tony

---

From: Robin Whittle <[email-masked]>
Date: Thu, 28 Jan 2010 21:44:12 +1100

Short version:      In the absence of better documented evidence that
                    filtering of incoming ICMP PTBs is anything more
                    than sporadic and quickly corrected, I can't
                    believe what Tony wrote.

Hi Tony,

> enterprises blocked all inbound ICMP when DDoS attacks started happening.

So a company XX does this, and to the extent that:

   1 - Any host in its network initiating a session involving "long"
       packets being sent (the TCP layer or application tries to send
       as long a packet as it can), where there is a lower PMTU
       outside XX's network than within, on the path to the
       destination host, will find RFC1191 PMTU unusable and will
       therefore be unable to communicate - unless the application
       backs off the packet length due to this failure.

   2 - Any host outside XX's network which initiates the session
       and which involves XX's host sending "long" packets will
       similarly be unable to communicate unless the XX hosts have
       their own, non-RFC1191 approach to reducing packet size.

It seems like 1 would be immediately noticed within XX and presumably
lead to an end to the filtering of incoming PTBs.

2 could be trickier for XX's administrators to notice, since those
who are most likely to notice it are scattered far and wide - unless
it is noticed by whoever runs the XX side of these communications.

Where is the occurrence of this documented in anything more than the
anecdotal way you describe it above?  I don't see any reason why it
would always be hush-hush.

>From what you wrote, I understand you regard RFC 1191 and RFC 1981
PMTUD to be so unreliable that an alternative must be developed and
implemented on all hosts.

Assuming this problem has been around for a while, then why wasn't
there greater support for RFC 4821?

In the time the PMTUD WG existed a handful of people exchanged about
a hundred emails.  A standards track RFC emerged and it is the only
thing around which can solve the problem you refer to, other than
applications backing off packet lengths themselves.

Maybe many UDP-based applications do this already - locally doing the
same thing that RFC 4821 suggests, but without sharing any
information with other packetization layers.

But TCP is a protocol which frequently is ready to send "long"
packets - as long as its local MSS allows.  It is reasonable to
assume that essentially no TCP sessions today currently use RFC 4821
PMTUD, since the only recorded implementation of it is a minimal one
in the Linux kernel, which is turned off by default.

So on one hand, you are telling us this PTB filtering happens often
enough to be a significant problem and is not going to go away even
though it must be causing some noticeable communication failure for
the networks which do it.

On the other, I see there is no alternative PMTUD mechanism in use
for TCP exchanges.

Yet if what you say is true, then the large TCP-dependent subset of
Internet communications must be going to hell in a handbasket right
now, as we speak, and the only hope of fixing it is by implementing
RFC 4821 PMTUD on every host, if only for TCP.

I look around and observe that TCP always seems to work for me and
everyone I know - and these TCP sessions are frequently using the
biggest packets the RFC 1191 PMTUD-controlled stack allows.

If what you wrote is true, then why isn't there a much greater
interest in PMTUD beyond RFC 1191 / 1981?  Why didn't a large number
of people work on a much better RFC 4821 than we have now - which
merely advises what should be done, without giving any framework for
how it should actually be implemented in the stack and in apps, with
the requisite API?

Even Matt Mathis http://staff.psc.edu/mathis/ writes:

  MTU is still a huge bottleneck, but this project is on hold.

He seems primarily concerned with using packets longer than 1500 bytes.

I just read the archives of the MTU mailing list - 12 messages since
2005.  There was only cursory mention of the filtering problem you
insist is so real and significant.

I just scanned each message in the archives of the IETF PMTUD WG:

from August 2005 to its closure in 2008.  I saw occasional references
to black holes (presumably due to the filtering you mention) but no
references to how prevalent this was.

Where is the evidence that PTB filtering is ever more than a
transitory, mistaken, condition?

  - Robin

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 28 Jan 2010 07:00:47 -0800

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Robin Whittle
> Sent: Thursday, January 28, 2010 2:44 AM
> To: RRG
> Subject: Re: [rrg] SEAL critique, PMTUD, RFC4821 = vapourware
> 

A recent accidental block was discussed in the thread
http://www.gossamer-threads.com/lists/nsp/ipv6/20779, where Google
was accidentally blocking ICMPv6 PTB.

I have heard of many similar problems where ICMPv6 PTB is being 
blocked; these take hours and occasionally days to resolve.  
Some of those can be attributed to growing pains of IPv6,
though -- perhaps a too-low limit for ICMPv6-per-second
from a router, and that just needed to be increased, or an overly-
aggressive ICMPv6 filter.

All modern routers allow limiting the number of ICMPs per second 
they send.  So if there is some reason they're sending a lot of ICMPs,
such as during an attack or when the interface is full of non-attack
traffic, they won't be able to send a legitimate PTB.  Forwarding real
traffic is more important than filtering ICMPs.  

The Cisco command is "ip icmp rate-limit unreachable", Juniper's is 
icmpv4-rate-limit.  These limits are actively discussed on operations 
lists, which implies those limits are actively used on the Internet.
 
> information with other packetization layers.

draft-petithuguenin-behave-stun-pmtud does it for UDP.  I know one 
video application that plans to use the technique described in 
that I-D.

> packets - as long as its local MSS allows. 

And fixing PTMUD is often done by tweaking the MSS.  Cisco 
equipment has long supported that functionality (and it was
mentioned as a quick fix in the thread I cited above).  This
"fixes" PMTUD failures.  Cisco command is "ip tcp adjust-mss",
Juniper commands are 'set flow all-tcp-mss' and 'set flow tcp-mss'.

> It is reasonable to

Sounds like you want a research paper?

-d

---

From: Robin Whittle <[email-masked]>
Date: Fri, 29 Jan 2010 13:32:04 +1100

Short version:     Ben Stasiewicz's research into significant
                   (33/404 = 8%) levels of PMTUD failure in IPv6 -
                   are tunnels causing a lot of this?

                   Geoff Huston's investigation of an IPv6 problem
                   presumably caused by one or more tunnels.

                   Tunnel ingress routers have a tough time
                   generating valid PTBs - I think they would need
                   to cache the initial parts of the packets
                   before they were encapsulated in order to be
                   able to generate a valid PTB to the sending host
                   if a PTB came back from a router in the tunnel
                   due to an encapsulted packet exceeding next-hop
                   MTU.

                   PTBs not making it correctly through NAT to the
                   sending host?  Google finds Norton Security on
                   a PC blocking IPv6 PTBs.

                   None of this makes me think that PTB-based
                   PMTUD must be abandoned and completely replaced
                   with end-to-end PMTUD, such as RFC 4821.

Hi Dan,

> Sounds like you want a research paper?

Yes.  Fortunately Matthew Luckie wrote to the MTU list, pointing
to the work of one of his students - Ben Stasiewicz:

  http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000708.html
  http://www.wand.net.nz/~bss7/pmtud-ben-final.pdf

This concerns a test of IPv6 websites.  0.11% of the Alexa top
million websites had an IPv6 version.  PMTUD problems were found to
a significant degree (figures quoted below), and it seems that some
servers had been configured to use a small packet size to avoid
such problems.

There's a discussion:

  http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/date.html#708

> lists, which implies those limits are actively used on the Internet.

OK - thanks for pointing to this Google incident.

What you describe doesn't seem to me to be a fundamental problem
with RFC 1191 / RFC 1981 PMTUD.  If there was a fundamental
problem, I would expect it to appear frequently in IPv4, and not
only affect PTBs but other ICMP messages, though I guess the PTBs
are the ones which would cause most trouble if they were either
filtered out, or not sent in the first place.

My understanding of Tony's message:

  http://www.ietf.org/mail-archive/web/rrg/current/msg05820.html

is that he argues that that the PTB approach to PMTUD - and indeed
anything which relies on ICMP messages - is iretrievably broken,
and so new end-to-end protocols would be needed, such as RFC 4821.

But I an yet to read anything which convinces me of this.

OK.

> Juniper commands are 'set flow all-tcp-mss' and 'set flow tcp-mss'.

If PTBs are either being dropped due to filtering, or not being
generated (such as due to competition with other ICMP messages
during an attack) and this is causing a significant problem with
PMTUD, then it seems the possible solutions are:

  1 - Stop the filtering and configure the routers so (within
      practical limits) attacks don't stop the generation of
      PTBs.

  2 - Adjust down MSS on sending hosts to fix the PMTU problems
      for TCP - but not other protocols.

  3 - Have TCP and other packetization layers in the stack and
      applications no longer reliant on PTBs, but implementing
      RFC 4821 fully, or performing similar end-to-end PMTUD
      independently, as you noted above was being done for a
      video app.

1 would clearly produce the best outcomes.

Benedikt, at the end of http://www.gossamer-threads.com/lists/nsp/ipv6/20779
strongly objected to fixing the MSS at a defensively low value.

Likewise Daniel Griggs:
  http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000714.html

Geoff Huston thinks the problem may be caused by tunnels or PTBs
not getting back to sending hosts behind NAT:

  http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-October/000721.html

I hadn't thought of NAT boxes . . . .  The NAT box would need to
look into the PTB, find the part of the original packet - IP header
and next 8 bytes at least - and then figure out which host behind it
to send the PTB to.  Presumably any self-respecting NAT box would
do this.  However, perhaps if actual PTB usage is minimal due to
manual or default low MTU/MSS settings, then perhaps there could
be wide enough adoption of NAT boxes which don't do this, before
anyone notices the problem sufficient to cause the developers
to add proper PTB handling.

Here is a Google report on "Norton Internet Security was disabling
all inbound ICMPv6 traffic not part of an existing session (i.e.
traffic that was not part of a response to outbound traffic
initiated by the host).":

  http://sites.google.com/site/ipv6center/icmpv6-is-non-optional

On tunnels, Geoff Huston wrote, in part:

   This additional header overhead implies that the tunnel's
   MTU is smaller than the "raw" interface MTU. The second
   problem with a tunnel is that there may be further tunnels
   "inside" the tunnel, so that the tunnel ingress is not
   necessarily aware of the true tunnel MTU. The third problem
   is that the routing of the interior of the tunnel may change,
   so that the tunnel MTU may be variable.

But for a tunnel to support PMTUD, I think the entry router needs
to maintain some kind of cache of recently sent packets so it can
construct a valid PTB to the sending host, when an encapsulated
packet in the tunnel hits an MTU limit.

This could be very expensive for large tunnels.  How long is the
fragment of each packet to be held?

   But if there is a condition that prevents the source from
   receiving packet-too-big ICMPv6 messages then the algorithm
   fails, and the application may hang when full-sized TCP packets
   are passed through the network. In some cases this may happen
   at a point well distanced from the two endpoints of the TCP
   session, so that the ICMPv6 filtering may be occurring at a
   point that is not under the control of the source or the
   destination.

   This is the basic reason why so many web server systems are
   averse to configuring themselves as dual stack IPv4 and IPv6
   servers. The problem is that through no fault of their own in
   the local configuration of the IPv6 server, and through no
   fault in the configuration of the IPv6 client, there are
   situations where the application fails, even though every part
   of the system appears to be functioning.

The first item on Geoff's list of solution is to fix the problem
in the network - whatever is dropping or failing to generate PTBs.

He contemplates changes to clients and servers, and in the end,
for practical reasons, to get his dual-stack server working
reliably, he drops the MTU to 1400:

  Waiting for every filter in the Internet to do the right thing
  with ICMP messages may well be a fruitless task, and adding
  further complexity into applications or the TCP protocol
  behaviour seems to take the long way around the problem. The
  most effective approach appears to be the simplest one as well
  - whether you are a dual stack client, or a dual stack server,
  the best way to get more reliable service under these rather
  strange corner cases is to drop your MTU.

This gets things going for now, but allows bad tunnels, PTB
filtering etc. to persist without complaint.  If Geoff Huston
lacks the time and inclination to figure out where the errant
tunnel is and provide the necessary feedback to its
administrators - who is going to do it?

So do we pull our horns in, wind back MTU/MSS values to
something timid and safe - and then forever be locked into
these limits even when most of the paths to other hosts around
the world support 9kbyte jumboframes?  This is just hobbling and
blinkering our hosts.

Tony seems to think we should give up on ICMP PTBs and expect OS
and app developers to implement RFC 4821.

I think that networks which filter out PTBs should change their
ways.

If there are tunnels not supporting PMTUD, then these need to be
fixed or changed.

Assuming IPv6 becomes much more widely adopted, these tunnel
problems will tend to go away, since I assume they are mainly
IPv6 over IPv4 tunnels.  So I think the prevalence of PMTUD
problems in IPv6, assuming it can be largely attributed to bad
tunnels, is no reason to abandon PTB-based PMTUD.

Ben Stasiewicz found:

  371 PMTUD_SUCCESS           OK    371
  214 RX_TOOSMALL
   22 RX_NODATA       ----]   Fail   33
   13 TCP_NOCONN          ]
    9 PMTUD_FAIL      ----]
    5 TCP_RST             ]
    2 RX_NOACK        ----]
    1 TCP_ERROR

The 214 servers classed as "RX_TOSMALL" were those where the
HTTP server returned packets too small to perform the PMTUD
test.  Perhaps this was a tweak to avoid PMTUD problems which
are apparently not uncommon (33/404) in IPv6 at present.

But what is the incidence of such troubles in IPv4?

  - Robin                   http://www.firstpr.com.au/ip/ivip/

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 29 Jan 2010 12:54:50 -0800

Robin,

> Sent: Wednesday, January 27, 2010 7:26 PM
> To: RRG
> Cc: Templin, Fred L
> Subject: SEAL critique, PMTUD, RFC4821 = vapourware
>

Yes; thanks for the correction. RFC4821 concerns packetization
layer path MTU discovery, where the application itself is a
packetization layer when, e.g., UDP is used as the transport.

> ITR on a long latency link.  (TTR Mobility complies with these

It seems then that the ITE and ETE would need to engage in
a guessing game based on packets *not* arriving within a
certain time? And, both the ITE and ETE have to maintain
a separate timer for every tunnel they belong to? I have
gone down that path many times in the past, and each
attempt invariably hit a dead end. You can check the I-D
archives for some of my expired works if you are interested.

> IPTM approach is arguably lighter weight than your SEAL approach.

With SEAL-FS, the ETE does not carry all the fragments.
Instead, it uses first-fragments for reporting purposes
only and otherwise discards all fragments.

> reliably delivered without MTU problems.

SEAL doesn't hunt back and forth. In SEAL, every data
packet is an implicit probe, and the ETE uses IPv4
fragmentation as an indication that it needs to tell
the ITE to reduce the size of the packets it is sending.

> which results from those goes to the sending host, which will not

If the source address of the original packet also
goes as the source address of the outer packet, then
wouldn't that constitute mixing both EID and RLOC
addressing within the same routing region? I thought
the whole purpose of the CES approach was to keep the
EID and RLOC routing and addressing spaces separate.

> How would this work for SEAL?

Using the UDP/TCP checksum as a nonce requires that the
ITE cache copies of its recently-sent packets. But then,
it would need to do this for every tunnel it belongs to
and it has no way of knowing for how long it will have
to retain the cached copies. With SEAL, the ITE never
has to cache packets in order to match them up with
any PTB feedback.

> bits being a nonce.  That would always be returned in any PTB.

SEAL uses the 32bit ID (gotten from the 16bit IPv4 ID
concatenated with SEAL's 16bit ID extension) as a nonce.
There is no need that I can see for including an
additional nonce.

> around with a protocol such as SEAL or IPTM.

In my understanding, in the interdomain routing region of
the Internet there is no close coordination regarding the
way "the network" is administered. There is also a wide
variety of network vendor equipment deployed in the
Internet which may have widely varying default behaviors.
So, in general it seems overly optimistic to assume that
all of the diverse policies, implementations and operational
practices out there could be brought into strict uniformity.

Fred

---

From: Robin Whittle <[email-masked]>
Date: Sun, 31 Jan 2010 05:01:09 +1100

Short version:     Mainly explaining my understanding of how
                   SEAL does PMTUD when used as part of RANGER
                   - when RANGER is acting as a Core-Edge
                   Separation solution to the routing scaling
                   problem.

                   Both RANGER and SEAL can to other things
                   besides this, and I find it hard to envisage
                   the subset of their operations which would
                   be used in a Core-Edge Separation setting.

                   I still thing it is wrong to develop a complex
                   and difficult protocol such as RFC4821,
                   because of some badly designed tunnels which
                   don't generate PTBs or due to a few networks
                   which filter out PTBs.

                   There's very little sign of anyone wanting to
                   develop or use RFC4821 - so I guess the
                   need or desire for it isn't be very strong.

Hi Fred,

> packetization layer when, e.g., UDP is used as the transport.

OK.  My understanding is that in the stack there is, for each
destination address, an MTU number which can be read and written by
any packetization layer.  The TCP packetization layer in the stack is
the most obvious one, but I guess if there was an SCTP layer there
too, this would also read and write this variable (or is it multiple
variables for each destination address?).  Then, for as many
applications have packetization layers, they would be able to read
and write these variables too.

> only and otherwise discards all fragments.

OK - thanks.

> SEAL doesn't hunt back and forth. 

4.3.9.1.2 mentions an "iterative searching strategy" - which sounds
like a fancy term for "hunt"!  This occurs only in IPv4 when the ETE
gets a first fragment shorter than 576 bytes, then this is
interpreted as a "runt fragment" and so is not regarded as a true
measurement of the limiting MTU.

> In SEAL, every data

To function correctly in IPv4, SEAL also relies on the first fragment
arriving at the ETE, and that fragment being the length of the
shorted MTU between the ITE and the ETE.  There's not much chance of
the first fragment being lost, so this is fine.

As far as I know, IPv4 routers are not required to make their
fragments (all but the last usually) the length of their MTU limit,
but I guess most of them do.

Assuming they all do, your IPv4 fragmentation approach has an
advantage where there are two or more MTU limits in succession.

If the first limit is 1400 and the second 1300, then by relying on
PTBs, as you do with IPv6, then first of all, the ITE discovers the
1400 limit, and sends a PTB to the sending host (SH) with the correct
value for that: 1400 - 24 (IPv4 header + SEAL header) = 1376.  The SH
tries again, and the encapsulated packet generates a PTB at the
router with the 1300 limit.  This is propagated back to the SH in the
same way, with a PTB with a 1276 byte MTU - and then all is well.

With your fragmentation approach, the ETE should get a first fragment
of length 1300 bytes.  It reports this back to the ITE, and it sends
a single PTB to the SH, with a value of 1276 bytes.

Which would be faster depends on the circumstances - since the PTB
back to the ITE will often be faster than the longer path all the way
to the ETE.

If the limiting router sends a fragment of length less than the
limiting MTU, then the SEAL ITE would adopt an unrealistically low
MTU value.  I guess this is unlikely.

IPTM doesn't hunt back and forth.  When a long enough packet arrives
at the ITR for the encapsulated length to fall between its two
current markers for the Zone of Uncertainty (low and high water
markers, I think, are the terms used in other protocols) the dual
packet probe technique will usually result in reliable new
information in most instances, due to one of:

  1 - The traffic packet is delivered to the ETR and the ITR is
      informed about this - so the ITR raises the low marker.

  2 - The traffic packet was not delivered and the ITR gets a PTB.
      This enables the ITR to lower its high marker and send a
      PTB to the SH with the correct MTU value to cause the SH so
      subsequent packets it sends, once encapsulated, will fit the
      MTU limit exactly.

  3 - If the long packet does not arrive at the ETR, but the short
      one does, and the ITR receives no PTB, then perhaps the long
      packet was merely lost.  However, if this happens repeatedly
      then the ITR should discern that there is an MTU limit below
      this length, and so adjust its upper marker downwards.  This
      will result in a PTB going to the SH, which will send
      shorter packets.  Over multiple iterations, the ITR will
      discover the MTU.  If the steps downwards taken by the ITR
      in reducing its upper marker are small, this will take a long
      time.  If they are larger, the true MTU will be found faster,
      but there may be more overshoot, resulting in somewhat
      smaller MTU for all hosts whose packets are tunneled to this
      ETR than is really needed.

      Once a pair of probe packets does arrive at the ETR, the ITR
      is informed of this by the ETR and it raises its upper marker
      accordingly

 Pretty quickly, the two markers would reach the same value and the
ITR would have a reliable measure of the MTU to this ETR.

The most likely pattern would be this:  Initially (before the ITR has
ever sent packets to this ETR address) the markers are wide apart.
Then the SH sends a packet which longer, once encapsulated, than the
low marker and also longer than the actual tunnel MTU, minus
encapsulation overheard.  So the the ITR does the two packet probing
protocol and gets a PTB.  Assuming this is from the router which sets
the lowest PMTU limit in the tunnel, then this PTB enables the ITR to
send a PTB straight back to the SH.  This also enables the ITR to
lower the upper marker to match the MTU reported in the PTB it just
received.

The SH will create another packet of the correct length to suit the
tunnel (after encapsulation it will be the length specified in the
PTB from the limiting router).  The ITR gets this and because it
would be longer, after encapsulation, than the low marker, the ITR
again sends it to the ETR with the two-packet tunneling protocol.
This time, it gets through, and the ETR reports this.  As soon as
this report gets to the ITR, the ITR adjusts its lower marker to the
same value as the upper marker - so there is no more Zone of
Uncertainty.

The SH will generally continue to send packets of this length or
shorter.  If some other SH (or another application in the same SH)
sends a packet which needs to be tunneled to the same ETR and which
is longer than this now reliably known PMTU value, then the ITR will
drop it and send back a PTB, without trying to send it into the
tunnel.  That application or SH will then send packets of the right
length.

I haven't figured out every detail of this - at some stage I intend
to work on it more and write it up as an ID.  For now, it is at:

> EID and RLOC routing and addressing spaces separate.

The "Separation" means that a subset of the global unicast address
space is used as "SPI" space (Ivip) or "EID" space (LISP).  This is
not a separate namespace, just a subset of the global unicast address
space, in the form of multiple DFZ-advertised prefixes which have no
name in LISP, but which are called Mapped Address Blocks (MABs) in Ivip.

The ITR tunneling uses the sending host's source address as the outer
source address for all ordinarily tunneled packets.  This enables any
ISP BR filtering - which drops incoming packets due to them having a
source address from any one of the ISP's prefixes - to be enforced on
the inner packet by all ETRs, by the simple method of dropping any
inner packet whose source address is different from the outer
header's source address.

This functionality is also enforced with the 2 packet probing
arrangement - the short A packet is also sent with its outer source
address being that of the sending host.

When a packet would be, after encapsulation, a length within the
"Zone of Uncertainty" then the ITR uses the special long (B) and
short (A) protocol.  The B packet's outer source address is that of
the ITR - so the ITR would normally get any PTB which arises from the
B packet.

The "sending host" address could be a conventional (non-SPI) address
  such as from a host on PA or PI space - or it could be on an SPI
address.  This is the source address of the packet the ITR is
processing.  Whether that is the actual address of the sending host,
or just that of a NAT box which the sending host is behind, is not
known to the ITR.

All hosts and most routers make no distinction between addresses in
the SPI subset and the rest of the addresses in the global unicast
address range.  Only ITRs treat them differently - and then only when
they appear in the destination address of a packet which is forwarded
to them.  Instead of forwarding the packet according to its
destination address, the ITR's FIB processes the packet differently.

If the ITR's FIB (which may be all in software, since the ITR may be
in the sending host or be implemented on a COTS server) already has
mapping for a micronet which matches this SPI address, it uses that
mapping (a single ETR address) to tunnel the packet.  If not, it
buffers the packet, requests mapping from a nearby QSD (full database
query server), installs the mapping (a micronet start and end
address, with a single ETR address) in its FIB and then tunnels the
packet accordingly.

There's no concept of "routing region" in Ivip.  ITRs in any place
would do exactly the same thing.  All other devices - hosts and
ordinary routers - make no distinction between SPI addresses and
remainder of the addresses in the global unicast address range.

> ITE cache copies of its recently-sent packets. 

The above procedure is only for the long (B) packet when the ITR is
still uncertain of the PMTU to a given ETR and the packet, if
normally encapsulated, would be of a length within this Zone of
Uncertainty.  The B packet is the same length, but is sent with the
ITR's address in the outer header.  Like the short A packet, it is a
UDP packet with special headers.  All the normally encapsulated
packets are IP-in-IP, with no other headers, and with the sending
host's address in the outer source address.  So normally encapsulated
packets can't generate PTBs to the ITR - only to the sending host,
which would not recognise them, since they arise from an encapsulated
packet.

So only for these traffic packets for which the ITR is using to
generate the B and A probe pair does it need to cache enough of the
initial packet to be able to generate a valid PTB to the sending
host.  The ITR would also cache the nonce, which it uses to secure
the ETR's response to these packets.

It would also cache the UDP header of the B packet, which includes
the checksum which is almost impossible for an attacker to guess due
to its dependence on the nonce.  I think the combination of an
unpredictable 16 bit ID in the outer header of the B packet, and the
influence of the nonce on the 16 bit checksum, would be sufficient to
prevent attacks succeeding at a significant rate.  If that wasn't
enough, the ITR could use 16 bit randomization of its UDP source
port, and randomize 8 or more bits of the destination UDP port too.

> But then,
> to retain the cached copies. 

As I noted above, the ITR doesn't cache its ordinarily encapsulated
packets.  The PTB would not go to the ITR.  It only needs to cache
the start of those packets it is sending with the two-packet probing
protocol.

If the ITR estimates the PMTU to a given ETR, and gets it right - and
then later the MTU falls, then there is a difficulty.  The normally
encapsulated packets which are too long will be dropped, the ITR will
not get any PTBs and the sending host will not recognise the PTBs
which are sent to it.

I can think of two approaches to minimising the impact of this.

One is to have the ITR periodically, such as every 30 seconds, send a
packet which (once encapsulated) is at, or close to, the MTU limit,
as a B and A probe pair.  This is assuming the ITR is continually
sending long packets to this ETR.

This will normally deliver the packet fine, and the ITR will be able
to confirm that the PMTU has not become any less than what it
assumes.  If the packet doesn't arrive, or if the ITR gets a PTB,
then perhaps the MTU has dropped and the ITR can find out what it has
dropped to.

It will usually find out the new PMTU from a PTB, but if there is no
PTB, then it will need to send more probes of various lengths until
one size does get through to the ETR.

It would do this as described above, without having to generate
special probe packets, by lowering its upper marker for the MTU
estimate by some value, such as 8 or 16 bytes, sending a PTB to the
sending host (or multiple sending hosts as they send packets which
would exceed this length, once encapsulated) and allowing the sending
hosts to create traffic packets which the ITR will send using the 2
packet probing technique.  This will rapidly reduce the value of the
upper marker to being equal to, or somewhat less than, the real PMTU
limit.  This will drag down the lower marker too.

The following second approach would only work if the returned part of
the packet was long enough to show the SH that it really did result
from a packet the SH sent.  In practice, this should always be the
case with IPv6, and I have been gained the impression that it is
common for IPv4 routers to send back more than the bare RFC1191
minimum anyway.  This would require the sending host to have a
modified stack which was ready to analyse PTBs which resulted from
packets in the ITR to ETR tunnel.  Assuming this modified stack code
could verify that the PTB was genuine, it would compute a new MTU for
this destination address, by subtracting the encapsulation overhead
(20 for IPv4, 40 for IPv6) from the MTU value in the PTB.

This one PTB would not help the SH learn about a reduced PMTU to
other SPI destination addresses it was sending to which also were
being tunneled to the same ETR.  But the same code would receive PTBs
from those packets too.  The ITR would be none-the-wiser, so the
first technique and the one below would still be important - but this
optional host upgrade would enable the SH to respond immediately and
correctly to a reduction in PMTU to an ETR.  Other applications in
other SHs would need to repeat this exercise, since the ITR doesn't
know these PTBs are occurring.

After 10 minutes, a sending host is allowed (RFC1191 / 1981) to try
sending a longer packet than was allowed by a previous PTB.  Then,
the ITR needs to recognise the time which has elapsed and use this
with the B and A probe technique.

This may be a little complex, considering multiple applications in
one sending host, multiple hosts and multiple destination SPI
addresses may all result in packets being sent to the one ETR address.

I can see ways of coping with this stuff - but some of it requires
carefully designed algorithms and considerable logic and state in the
ITR.

If we can upgrade the DFZ and other routers with firmware, then all
this encapsulation and PMTUD stuff can be ignored - by using Modified
Header Forwarding instead.  That is the way to do it in the long-term
future, even if we start with encapsulation.

> With SEAL, the ITE never

Yes - here's my understanding of how your SEAL ID specifies how the
ITE and therefore the SH perform PMTUD in the tunnel path, between
the ITE and the ETE.

I assume in all cases that the ITE initially sets S_MRU for this ETE
to "infinity" as described in 4.3.3, and then uses one of the
following methods to reduce it.  This is me trying to imagine how the
SEAL would be used for ITE to ETE tunneling when RANGER is used as a
Core-Edge Elimination architecture.

RANGER can be used for many more purposes than this, and so can SEAL,
so it is quite a challenge to decide which parts of the IDs to
ignore.  I understand that in this application, the ITE will be
reducing the S_MRU value for each ETE it tunnels to.  I think that in
other SEAL applications, this may not occur, and so you have
arrangements for using SEAL segmentation to send long traffic packets
as multiple SEAL-segmented packets to the other end of the tunnel.
But this would never be invoked in a Core-Edge Separation
application, since the ITE always sends PTBs to the SHs to have them
reduce their packet length.

In this application of SEAL, I understand there is no need for any
mid-layer protocol between the IPv4 or IPv6 header and the SEAL
header, or between the SEAL header and the traffic packet.  This is
not clearly specified anywhere, since the SEAL and RANGER documents
are general purpose, and their use for a scalable routing solution as
a Core-Edge Separation architecture is only one thing they could be
used for.

Firstly I describe my understanding of what your ID specifies for IPv4.

Secondly I describe two other ways you might do PMTUD with IPv4,
without using DF=0 packets.  These would avoid whatever risk there
might be of setting the ITE's PMTU estimate too low due to a limiting
router sending out fragments which are shorter than the limiting next
hop MTU.

Finally, I describe my understanding of what your ID specifies for IPv6.

This is partly for my own reference, since it took me many hours to
discern this by reading the SEAL ID and corresponding with you.

IPv4:

  The ITE sends a DF=0 packet into the tunnel.  This starts with
  an IPv4 header, then has a SEAL header (there's no mid-level
  protocol in this Core-Edge Separation usage of RANGER and SEAL)
  and then the inner packet, the original IPv4 traffic packet.

  The source address in the outer header is that of the ITE and
  the destination address is that of the ETE.  The 32 bit
  SEAL ID is split in two.  16 bits go into the IPv4 header's
  ID field and 16 into the SEAL header's ID Extension field.

  The limiting router in the tunnel (the one where the next-hop
  MTU is less than the the length of this whole packet) fragments
  it into at least two fragments.

  Now the second para in 4.4.2 comes into play:

        When the ETE processes the IP first-fragment (i.e.,
        one with MF=1 and Offset=0 in the IP header) of a
        fragmented SEAL packet, ...

  The first para was for reassembling packets which had been
  fragmented by the SEAL protocol.  But the second para is for
  SEAL packets, as was just sent, being fragmented by a
  router between the ITE and the ETE.  This only occurs for
  IPv4 and I think it would be helpful to mention IPv4 in this
  paragraph.  Maybe it needs its own section.

       ...  it sends a "Reassembly Report - Fragmentation
       Experienced" message back to the ITE with the S_MSS field
       set to the length of the first-fragment and with the
       S_MRU field set to no more than the size of the reassembly
       buffer (see Section 4.4.5).

  I think this last part about the value of S_MRU is not clear
  enough.  What value should it be set to?

  I will assume it is set to some non-zero value.

  Assuming the limiting router sent out the first fragment with
  a length equal to the limiting next-hop MTU, then this MTU
  value is now in the S_MSS field of the message sent to the
  ITE.

  This message arrives at the ITE.  This message, according to
  Figure 4, contains:

       As much of invoking packet as possible without the
       message exceeding 576 bytes.

  Maybe your ID specifies this, but I am having trouble
  following it - there has to be a way the ITE securely
  accepts this "Fragmentation Experienced" message.

  As far as I know, the ITE looks into the message, finding
  the initial part of the packet which the ETE received as
  a first fragment.  That will contain the outer IPv4 header
  and the SEAL header, and from this these the 32 bit SEAL ID
  in the SEAL encapsulated packet can be found.

  I think you either cache the recently sent 32 bit SEAL IDs
  or maintain a sliding window function over their range so
  you can easily identify a value which was used in the last
  second or two.  In a given ITE, each ETE has its own SEAL
  ID counter.  Its value is intitialized randomly when the
  state for this ETE is created.  After that, its value
  increments with each each packet sent to the ETE.

  (I may adopt this incrementing value per ETR arrangement,
  with its sliding window, rather than using a nonce.)

  The wider the window in time, the longer you can accept these
  messages.  Since the ETE and the ITE could be on opposite
  sides of the Net, I guess you need to have a window which
  accepts SEAL IDs sent at least a second ago.

  The longer the window in time, and the more packets the
  ITE sends to this ITE, the wider the window is numerically
  and the easier it is for an attacker to guess a valid value
  and have the ETE accept a PTB with a low enough value
  to cause lost efficiency - for the next 10 minutes or so.

  Now to 4.3.9.1.2:

       4.3.9.1.2. Fragmentation Experienced (Code=1)

       If the value in the S_MRU field is non-zero, the
       ITE records the value in its soft state for this ETE.

  This means this value is stored in the S_MRU variable for
  this ETE, as defined in 4.3.3.  As noted above, I am not
  clear on what value was written into this field of the
  report by the ETE.

       The ITE then adjusts the S_MSS value in its soft state.

  This means this value is stored in the S_MSS variable for
  this ETE, as defined in 4.3.3, subject to the instructions
  in the next few sentences.

  I am a bit confused about the differing roles of these two
  variables.

       If the S_MSS value in the Reassembly Report is greater
       than 576 (i.e., the nominal minimum MTU for IPv4 links),
       the ITE records this new value in its soft state.

  OK - this is based on the assumption that the length of the
  first fragment received by the ETE reflects the limiting
  MTU of the ITE to ETE path.

       If the S_MSS value in the report is less than the current
       soft state value and also less than 576,

  How could the ITE's S_MSS value for this ETE be less than
  576?  I can't see how.  If it can't be, then the first part
  of the above sentence may be redundant.

       the ITE can discern that IP fragmentation is occurring
       but it cannot determine the true MTU of the restricting
       link due to a router on the path generating runt
       first-fragments.

  Then the next paragraph describes the "iterative searching
  strategy" to find the correct (or near enough, but perhaps
  lower) value for the S_MSS variable for this ETE.

  I think this paragraph is unclear.  I think it should state
  that the probes are occurring only due to traffic packets
  arriving at the ITE and being tunneled to this ETE, and
  these being long enough.  Since SEAL treats all packets
  as probes, this use of the term "probe" may be confusing -
  since in fact all packets may be probes.

  I think this paragraph should describe the process in detail
  - I guess it is only occurs with real traffic packets.

  The reference to section 5 of RFC1191 is not very helpful
  because it describes several algorithms.

IPv4 - my suggestion for doing it with DF=1 packets

  A - If you can be sure the routers send back more than
      the bare minimum IPv4 header + 32 bits.  (Its just
      firmware updates to have routers do this and maybe
      most or all of them already do.)

      Send the SEAL packets as noted above, but with DF=1.

      If there is an MTU problem, the ITE will get a PTB
      with the MTU value it needs, plus enough of the
      SEAL packet to extract sufficient of the traffic
      packet to make a valid PTB for the SH.

      The MTU value from the received PTB is written into
      the S_MTU for this ETE.  This gives an exact value
      without the problem of potential "runt packets" which
      arises with DF=0 in your current process.

      The ITR subtracts 24 from the MTU value it received
      in the PTB from the limiting router (20 bytes of IPv4
      header + 4 bytes of SEAL header) and uses this value
      in the PTB to the SH.

      This will work fine - the SH will then send packets
      of the correct size, so when they are SEAL encapsulated
      they will not have a problem with this MTU limit.

      If any other SH, or another application in the same
      SH, sends a packet whose length exceeds the new MTU
      value minus 24, then the ITE will send back a PTB
      accordingly.

      If there is a further, lower, MTU limit en-route to
      the ETE, then the above process will be repeated.
      This is similar in principle to your IPv6 approach.

  B - If you have to assume that some or all routers between
      the ITE and the ETE only send back the bare minimum
      amount of packet in their PTB, then you can still
      accept these packets securely, and calculate a proper
      MTU value to send to the SH, as described above.

      In order to be able to generate a valid PTB, you need
      the ITE to have cached the IPv4 header and the next
      32 bits which follows (24 bytes) for each packet sent
      which you think might give rise to a PTB.  You don't
      need to do this with packets shorter than some constant
      - depending on whatever is the lowest PMTU you ever
      expect to find between an ITE and an ETE.

      I guess you only need to cache these 24 byte items for a
      second or so.  You need to be able to index into the
      cache by using the 32 bit SEAL ID retrieved from the
      16 bit IPv4 ID and the 16 bit SEAL ID Extension in the
      initial part of the encapsulated packet, which is in
      the first fragment, as returned in the PTB.

IPv6:

  The ITE creates an IPv6 header and a Fragment Header.  As far
  as I know, there is no SEAL header.  I think this should be
  made more clear in the final part of 3.4.3.

  The 32 bit SEAL ID is written into the Identification field
  of the Fragment Header.  The ITE then appends the traffic
  packet.

  The result is forwarded towards the ETE.  If it is too big for
  a next-hop MTU in any router en-route to the ETE, that router
  sends back a PTB to the ITE with an MTU value, and enough of
  the original packet for the ITE to construct a valid PTB to the
  SH.

     (I can't find where your ID describes the reception of
      the PTB.  Section 4.3.8 should cover this, but makes no
      specific mention of PTB messages.)

  The ITE secures the acceptance of the PTB by using comparing
  the 32 bit SEAL ID, as noted for IPv4 above, via a cached set
  of recently used values or some kind of window function

  The MTU value is written to the S_MTU for this ETE.

  The ITE subtracts 48 from the MTU value (40 bytes for the IPv6
  header and 8 bytes for the Fragment Header) and uses this to set
  the MTU value in the PTB which is sent to the SH.

  As with the IPv4 approach, any packets arriving at the ITE
  which will be tunneled to this ETE, if longer than the MTU
  value minus 48, will be dropped and used to send a PTB to
  the SH.

  This is identical in principle to my IPv4 suggestion A above.

Returning to Ivip's IPTM protocol:

With IPv6, I could avoid caching any part of the packet if I could
rely on the ITR getting a PTB, since the PTB is guaranteed to contain
plenty of the inner packet - enough for the sending host to
recognise.  (Actually, the minimum amount of original packet returned
could be less than what should be returned to the SH, due to the
encapsulation IP, UDP and IPTM header which precedes it.  Still,
enough should be there that any SH should be able to recognise it.)

I still need to send two packets, since the long B packet does not
contain the full traffic packet.  It has extra things - a UDP header
and an IPTM header, with a 32 bit nonce.   The last part of the
traffic packet is not in the B packet

The part which doesn't fit is contained in the A packet.

If both A and B arrive at the ETR, the whole traffic packet is
delivered.  The A packet is matched to its B packet with the nonce
they both contain in their IPTM headers.

The ETR will send a message to the ITR, also secured by the nonce,
to tell it that both parts arrived.

If the B part doesn't arrive, after a ~0.5 second time-out, the ETR
will send a message to the ITR telling it that only the A part arrived.

If only the B part arrives, the ETR sends a message to the ITR to
that effect too.

With IPv4, if it could be assumed that all routers would return
sufficient of the packet to include the first 24 bytes of the inner
packet, then I could use the same approach as for IPv6 - and so avoid
caching any part of the traffic packet in the ITR.

If I couldn't assume this, then there are two approaches:

  1 - Do as for my suggestion B above - have the ITR cache the
      first 24 bytes of traffic packets used for this 2 packet
      probing technique.  The cache would be indexable via the
      nonce sent with each packet - and the caching time would
      be about a second.

  2 - Avoid caching in the ITR, by including the first 24 bytes
      of the traffic packet in the A packet.  If the A packet
      arrives at the ETR, and the B packet doesn't, the ETR
      can report this, and return the 24 bytes from the A
      packet.

The ITR is only doing this 2 packet probing technique infrequently,
so the caching approach is not particularly expensive.  Caching the
first 24 bytes of the packet has an advantage that when the ITR gets
a PTB - as would normally be the case if the B packet was too long -
then the ITR can send the PTB immediately, rather than waiting for a
message from the ETR, which would necessarily arrive at least a
second after the ITR tunneled the packet.

The detection of a PMTU limit doesn't have to be absolutely
bullet-proof.  It should not result in the ITR deciding that the PMTU
is lower than it actually is, but if, for some reason, the probe
process produces an indeterminate result - such as the ITR not
getting anything back from the ETR as a result of the A or B packet,
and no PTB either, then the ITR takes no further action.  This is
indistinguishable from ordinary packet loss.  The most likely outcome
is that the SH will try again, with a similar sized packet (unless it
is doing RFC 4821 ... which no hosts appear to be doing at present)
and the ITR will again generate the B and A packets.  Then, the most
likely outcome will be that the ITR learns something definite about
the PMTU, and so reduces its Zone of Uncertainty.

Doing this PMTUD stuff in the FIB of a big router handling gigabit
and 10 gigabit links could be quite challenging.  It doesn't have to
be done this way, since Ivip ITRs and ETRs can be implemented in
software on ordinary servers, which are inexpensive and can still
handle (I guess) gigabit traffic rates.  Also, having the ITR in the
sending host is a zero cost way of ensuring each ITR doesn't have to
juggle too many of these PMTUD probing sessions at once.

OK.

> practices out there could be brought into strict uniformity.

OK - but at some point we need to stop adopting band-aid measures
like artificially limiting MSS or MTU values.  That just lets the PTB
filtering and lousy tunnels be less noticed.   We should not be
trying to upgrade the stacks of all hosts in the world because a few
end-user networks filter PTBs or ISPs and perhaps end-user networks
run tunnels which don't support the otherwise perfectly good RFC 1191
/ 1981 PMTUD techniques.

We would just be heaping limitations and complications on ourselves
in an overly-defensive, expensive and inefficient attempt to cope
with failure of a few ISPs and end-user networks to run the Internet
as it needs to be run.  We are paying the ISPs.  The end-user
networks which are filtering PTBs are disrupting a subset of their
own communications.

I just think it wrong in principle to develop messy new protocols
such as RFC 4821 to cope with these failings.

  - Robin

---

From: "Eddy, Wesley M. (GRC-MS00)[ASRC AEROSPACE CORP]" <[email-masked]>
Date: Fri, 29 Jan 2010 15:22:34 -0600

>From: [email-masked] [mailto:[email-masked]] On Behalf Of
>Dan Wing
>Sent: Thursday, January 28, 2010 10:01 AM
>To: 'Robin Whittle'; 'RRG'
> ...

Here are three for Robin's sake; this issue is well known to the
transport layer folks:

http://www.icir.org/tbit/tbit-Aug2004.pdf

http://www.caida.org/workshops/wide/0603/slides/mluckie2.pdf

--
Wes Eddy
MTI Systems