Subject: [RRG] TRRP-Via - option header vs. encapsulation

From: Robin Whittle <[email-masked]>
Date: Tue, 18 Sep 2007 00:45:57 +1000

On 24 August, in message:

  http://psg.com/lists/rrg/2007/msg00293.html

Bill Herrin wrote, in part:

>> Robin Whittle wrote:
>> They all have similar problems with tunneling overhead,
>> fragmentation, breaking traceroute and Path MTU Discovery.
>
> Hi Robin,
>
> What about something like this:
>
>  http://bill.herrin.us/network/trrp-via.html

I have only a partial understanding of your goals here, and what
your proposed mechanism involves.  I have almost no prior
understanding of loose source routing, and found this explanation
helpful:

  http://www.synacklabs.net/OOB/LSR.html

I understand that (for IPv4) your proposal involves the ITR adding
an option header to it which functions similar to loose source
routing -  rather than adding a second outer header (encapsulation,
to tunnel the packet).

I understand your proposal is like loose source routing, but with a
different option number (0x9E = 158) and with the following
important differences:

      Loose Source Routing          Via IP option

1     Destination address           Destination address
      is changed to be that         remains that of the
      of the intermediate           original packet - the
      router.                       destination host.

2     The address of final          The address of the intermediate
      destination host is           router (ETR) is specified in the
      specified in the option       option header.
      header.

3     All routers en-route to       All routers en-route to the ETR
      the intermediate router       must find the option header,
      need only look at the         and use the address there rather
      ordinary destination          than the packet's DA - for
                                    routing.

I think the third point means that all DFZ routers (and maybe some
routers in ISP networks, depending on where the ITRs and ETRs are)
would need to be upgraded before your proposal could be used.  If
so, then I think the proposal is impractical, since this violates a
key requirement for incremental deployment.

I understand this proposal involves a lower additional number of
bytes, which is good, but I think that even a single byte extra
raises the whole ugly problem of fragmentation (at least in IPv4).

I don't know enough about ICMP messages to know whether the other
desired benefit would really occur - enabling Path MTU Discovery
between the ITR and ETR, with the sending host getting the messages
and adjusting its packet lengths accordingly.

Does the router generating the ICMP message back to the sending host
include just the basic IPv4 header and "8 bytes following", or is it
supposed to include the option parts of the header too?  If the
latter, the sending host will not recognise the option parts, since
the ITR added them - which would break PMTUD.

The MTU reported by that router (RFC 1191) would not help, since if
the sending host used it, the ITR's addition of the option header
would still cause the resulting packet to be too long.  You mention
this in your message.

It seems that PMTUD is not reliable anyway, depending on how deep in
the ISP etc. networks the ITR and ETR are located due (for instance)
to there being generally less filtering of ICMP messages in the DFZ
compared to inside ISP or end-user networks.

I want the ITRs and ETRs to be located potentially quite deep in the
networks, to provide flexibility and therefore large numbers of them
- to reduce the load on any one ITR or ETR.  ITRFH involves the
sending host (not behind NAT, but perhaps done in the NAT device,
which is not, in the case of DSL routers, a hardware based router)
doing a caching ITR function.

  - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: "William Herrin" <[email-masked]>
Date: Tue, 18 Sep 2007 09:52:52 -0400

>       router.                       destination host.

Robin,

Thanks for your comments. I had given up on the Via option in part
because of the need for all routers in the path to be upgraded to
support it but your observation about loose source routing resolves
that problem. I've revised the TRRP Via option as a result and posted
the new version here: http://bill.herrin.us/network/trrp-via02.html

> the ITR added them - which would break PMTUD.

My interpretation of the RFCs is that the router MUST return the
entire IP header including options, plus 8 bytes of payload. Those 8
bytes of payload will include the UDP or TCP port numbers if its a UDP
or TCP packet. The router MAY return additional payload up to what
will fit in a packet the size of IPv4's minimum MTU which is something
like 536 bytes.

Path MTU discovery is the other problem with the Via option. The host
can't be expected to understand the fragmentation needed message for
the altered packet and since the fragmentation needed message doesn't
return to the ITR, the ITR can't mediate. That's one reason I picked
GRE as the default encapsulation for TRRP instead of Via.

I tried to tackle PMTUD from a couple different directions in the
draft at http://bill.herrin.us/network/trrp-via02.html based in part
on Brian Carpenter's comments on how PMTUD works in IPv6. I'd be
interested in your comments; the approach may work for Ivip and LISP
as well.

And I now owe you a more careful read of Ivip. ;)

Regards,
Bill Herrin

-- 
William D. Herrin                  [email-masked]  [email-masked]
3005 Crane Dr.                        Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Robin Whittle <[email-masked]>
Date: Wed, 19 Sep 2007 02:11:17 +1000

Hi Bill,

I think your new proposal:

   http://bill.herrin.us/network/trrp-via02.html

looks very promising.  I haven't looked at the IPv6 section yet.

I think "next best preference ITR" should be "next best preference ETR".

I don't understand this:

  "If an ETR is unreachable, the via-tagged packet will follow the
   default route and fall back into an ITR."

Can you explain this more fully?

I think the first major requirement:

  ITRs and ETRs located so that there is always at least an 1280
  byte MTU on all routers between them.

is practical.

I am not so sure about this one, at least for Ivip:

  No filtering of ICMP destination unreachable messages between
  wherever the ITR and ETR are located.

because I want ITR functions to be able to be performed in sending
hosts, and I want ITRs and ETRs to be potentially quite deep in the
networks, not just at border routers.

By the way, I found a good explanation of how ICMP filtering can
screw up PMTUD:

  http://www.netheaven.com/pmtu.html

I want to think about TRRP-Via02 lot more, but I can imagine
adapting it to Ivip, by removing the second and subsequent lower
priority mapping options.

TRRP-Via and TRRP-Via02 are like Ivip in that the Source Address
(SA) of the packet when travelling between ITR and ETR is the same
SA as of the sending host.  All other approaches have the SA being
that of the ITR.

Your proposal doesn't have any means by which an aggrieved router
could find out which ITR sent the packet to it in error - as with
the current Ivip proposal.  I considered using UDP encapsulation so
I could include the ITRs address in the UDP data to solve this problem.

I could restructure your proposal (for IPv4 at least) to use and
encapsulation header rather than the modified original header plus
an option header.  This would be much the same as Ivip.  However, I
think your proposals about the ITR:

  Automatically assuming a shorter (say 1280) MTU and therefore
  fragmenting the packet before sending it to the ETR.

  Sending ICMP destination unreachable messages to the sending
  host in order to get it to send subsequent packets short
  enough not to need fragmentation.  (I am unclear as to whether
  one or the other or both of the above would be used.)

  Trying to find a higher MTU later, assuming there is more
  traffic to this ETR.

  Modifying Syn packets to make TCP run with short enough
  packets that they won't need to be fragmented to fit inside 1280.

are really interesting and probably practical.  So I will think
about adopting them into Ivip, either with an encapsulation header
as Ivip currently uses or with your new option header arrangement.

If I use your option header arrangement, I wouldn't need any list of
"been there" addresses, since there is no concept in Ivip of an ETR
acting like an ITR, to look up the mapping and to send the packet to
some other ETR according to second priority mapping information
(there is only one mapping option in Ivip) if the first ETR can't
reach the destination host.

If the option header I am thinking of for Ivip just included the
destination address of the ETR (4 bytes) and the 4 bytes which
precede it (the first 4 bytes of the option header), this is 8 bytes
- 12 bytes shorter than IP-in-IP encapsulation as Ivip currently uses.

If I added 4 bytes to carry the ITR's address, this would make it 16
bytes, which is 4 bytes shorter than plain IP-in-IP (which doesn't
have the ITR address, as I use it in Ivip) and a lot shorter than
using UDP encapsulation, as I previously proposed to carry the ITR
address.

Is there any reason to think that routers in general - BGP and
internal routers - will be upset by an option header, specifically a
new kind of option header as you suggest?

  - Robin

---

From: "William Herrin" <[email-masked]>
Date: Tue, 18 Sep 2007 13:59:19 -0400

Robin,

That's okay; there's nothing there to look at yet. :)

> I think "next best preference ITR" should be "next best preference ETR".

Fixed.

> Can you explain this more fully?

TRRP works by removing all but a few thousand "globally routable"
prefixes from the BGP table. The ETR's IP addresses are selected from
these globally routable prefixes. So if you change ISPs you keep your
CIDR block but get a new ETR address.

Any destination which is not in the globally routable space follows a
default route which leads to the nearest ITR. There it is encapsulated
(or in Via's case tagged) with an ETR as its destination address.

If the Via tagged packet reaches a router in the network which has no
route in the table corresponding to that ETR then the packet will
follow the default route and end up back at an ITR. That means the ETR
is unreachable so the ITR should try the next one.

>   wherever the ITR and ETR are located.

It was actually no filtering of echo request/response messages and
destination unreachable messages between the ITR and ETR. This is
because an ITR trying to discover a larger path MTU to the ETR will
assume the ETR is dead if it can't get a sensible response to a ping.

That requirement can be removed but the price is that the Via ITR will
have no way to independently determine that an ETR is unavailable.
It'd have to rely solely on the DNS route server.

> By the way, I found a good explanation of how ICMP filtering can

I had the pleasure of dealing with this on a regular basis back in my
ISP days. One of my upstreams was linked to a small stub of their
network. Their primary path was a T3 but their backup path was a GRE
tunnel through another provider. Every time that T3 went down I'd get
customer calls complaining that certain major web sites were
unavailable. The sites' TCP stacks would negotiate a 1500 byte packet
size with the customers but the sites' firewalls would drop all ICMP
packets on the floor so when they hit that 1470ish byte GRE tunnel
that was the end of the line.

I tell you, folks who think ICMP=Ping ought not be allowed to touch a firewall.

The standing order to customer support was: talk the customer through
adjusting the MSS value in their Windows Registry. I'm sure you can
imagine how well that turned out.

> I could include the ITRs address in the UDP data to solve this problem.

I considered adding that, but I'm not sure how much more useful to the
aggreived router it is to know where the ITR was than it would be to
know anything else about the other routers the packet passed through
to get there.

> new kind of option header as you suggest?

Another gentleman mentioned to me privately that currently deployed
routers tend to kick any packets with options to the slow (software)
forwarding path since the options might require the router to take
special action. Perhaps its possible to change that behavior with a
software upgrade that offers a configurable to ignore record-route and
timestamp options in favor of keeping IP packets with options in the
fast path.

Regards,
Bill

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 19 Sep 2007 08:25:38 +1200

...
> timestamp options in favor of keeping IP packets with options in the

Operators have been very leery of unrecognised or new options for
many years - in fact my own proposal to solve the R&A problem using
IPv4 options was shot down on those grounds at IETF 29.

Requiring an upgrade to every router in the Internet to recognise
(or at least, not to discard) a new option is a very tough deployment
strategy.

    Brian

---

From: Robin Whittle <[email-masked]>
Date: Wed, 19 Sep 2007 12:34:52 +1000

RW> Is there any reason to think that routers in general - BGP and
RW> internal routers - will be upset by an option header, specifically a
RW> new kind of option header as you suggest?
BH>
BH> Another gentleman mentioned to me privately that currently deployed
BH> routers tend to kick any packets with options to the slow (software)
BH> forwarding path since the options might require the router to take
BH> special action. Perhaps its possible to change that behavior with a
BH> software upgrade that offers a configurable to ignore
record-route and
BH> timestamp options in favor of keeping IP packets with options in the
BH> fast path.

Hi Brian,

> IPv4 options was shot down on those grounds at IETF 29.

Indeed - 13 years ago you were on the case:

http://mirror.switch.ch/ftp/doc/ietf/94mar/area.ipng.94mar.txt

    Most people felt that AEIOU would work and could, with effort,
    be developed into a viable stop-gap solution.  There was one
    significant technical issue, the impact of option analysis on
    local router performance.

http://mirror.switch.ch/ftp/doc/ietf/94mar/aeiou-minutes-94mar.txt

    It is pointed out that it is dangerous to assume that new
    options will survive passage across the Internet; Rob Ullman
    has done some investigation on what routers and hosts do to
    options.

    After this discussion, Brian is not inclined to propose it as an
    ''active proposal.''  No mandate for a working group is
    perceived, but it is worth keeping the discussion alive for
    anybody interested (send mail to Brian Carpenter if this is
    you).  Deering noted that AEIOU should go into the same status
    as as EIP: honored, revered, unimplemented.

Unless there is consensus that today's installed base of BGP and
internal network routers would handle IPv4 packets with a new option
header with the same ease and speed as ordinary IPv4 packets, I
propose that the option-header aspects of Bill's IP-Via proposal
also be "honored, revered and unimplemented".

So it is back to the drawing board with IP-in-IP, UDP or perhaps GRE
encapsulation.  However, while I need to think about it a lot more,
I still like Bill's ideas about ITR behaviour in fragmenting packets
initially, and being able to assume a 1280 MTU in the path to the
ETR.  Modifying SYN packets sounds good in some ways, but involves
deep packet inspection for lots of short packets (a good DoS target
too), and I think it is rather oppressive to have routers messing
with the contents of host's attempts to initiate communications.

  - Robin

---

From: Robin Whittle <[email-masked]>
Date: Wed, 19 Sep 2007 13:46:49 +1000

Hi Bill,

I wrote in a previous message why concerns about BGP routers'
ability to quickly handle packets with option headers, or at least
novel option headers, seems to make your TRRP-Via proposal impractical.

RW>   "If an ETR is unreachable, the via-tagged packet will follow the
RW>    default route and fall back into an ITR."
RW>
RW> Can you explain this more fully?
> CIDR block but get a new ETR address.

Wouldn't that mean there are only "a few thousand" ISP border
routers in the Net?   Perhaps you mean tens of thousands or a
hundred thousand or so.

> is unreachable so the ITR should try the next one.

OK - thanks for this explanation.  I need to reread your TRRP
proposal in its various pages - maybe this was clearly explained there.

> assume the ETR is dead if it can't get a sensible response to a ping.

OK.

> It'd have to rely solely on the DNS route server.

At one point, I recall you wrote that TRRP is like Ivip, and unlike
LISP and eFIT-APT, in that it does not integrate multihoming service
restoration into the protocol or the ITR functionality.

I reflected this in the comparison chart:

  http://psg.com/lists/rrg/2007/msg00291.html

This business of ITRs having a list of options for which ETR to use,
and ETRs acting like ITRs if they can't send packets to the
destination host makes me think that TRRP does in fact have a lot of
multihoming service restoration functionality built into it.

With Ivip, it is up to the end-user - manually or more likely with
their own multihoming monitoring system - to quickly update the
mapping database so all the ITRs are tunneling packets to the
correct ETR.

Thanks for your account of the trouble caused by sites filtering
ICMP messages in a way which screws up PMTUD.

> know anything else about the other routers the packet passed through

I agree, in general, except for the case where the ITR is a
legitimate ITR which is malfunctioning.  It is vital that ITRs, at
least for Ivip, take themselves off-line (or if they are ITRDs, at
least stop advertising the relevant address blocks) if they are no
longer receiving mapping updates.

I considered this in:

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-arch-00.html#rfc.section.14.1.2.3

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-arch-00.html#rfc.section.14.1.2.7

To aid in debugging errant tunneling, which is errant due to the
end-user entering into the mapping database the IP address of
something other than the correct ETR, I proposed a centralised
search system for the current state and recent history of the entire
Ivip mapping database.

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-arch-00.html#rfc.section.14.1.2.6

For instance, if some router (or host) gets a burst of tunnelled
packets, even for a second or two, the operator of that machine
needs to be able to find out who put that machine's IP address into
the Ivip mapping database.

 - Robin