Subject: RE: hop-by-hop and router alert options

From: Tschofenig Hannes <[email-masked]>
Date: Wed, 25 Aug 2004 16:33:53 +0200

hi pekka, 

it is interesting to hear your opinion on these issues. i would like to
point out a few things: 

- there have been a number of discussions about the bandwidth broker concept
in the past and the disadvantages are known. (my personal opinion) if it
boils down to the protocol details then the difference between the rsvp
architecture with a pep/pdp and a bandwidth broker is not hudge. the central
entity which performs the authorization (or policy) decision was already
mentioned. 

- from a signaling protocol point of view there is little difference between
path-coupled and path-decoupled signaling in many cases. even path-decoupled
signaling is path-coupled in some sense - you do not want to signal your
messages to networks where the data traffic will never flow through. one big
difference is the discovery mechanism. you are certainly right with your
observation that you could use a number of different discovery mechanisms to
communicate with devices along the path. if we can learn something from rsvp
with this respect then it should that multiple mechanisms for discovering an
node along the path has to be supported - not only router alert options. i
think such a decision would increase the number of usage scenarios.

btw, you might not know about the path-decoupled signaling (pads) bof (see
http://www.ietf.org/ietf/03mar/pads.txt) where people spoke about reusing
path-coupled signaling protocols in order to provide path-decoupled
signaling. 

- denial of service issues with the router alert option: it is true that
router alert option processing is slower than packets which do not have the
router alert option set. however, i am not sure whether this is truely a big
issue based on the most recent information i saw (see
http://www.welzl.at/research/projects/ip-options/).
furthermore, i always had the impression that other mechanisms even
introduce more denial of service risks. for example: if you have your
bandwidth broker example then you need to deal with dos issues against this
device as well (and there are plenty of them). basically, you need to
address denial of service issues at any place, regardless of the specific
solution. 

- firewall issues: you argue that an operator might not allow users to open
pinholes. why not? if you authenticate and authorize users then you can make
a decision which user is allowed to open a pinhole. furthermore, you already
do it today with more complex mechanisms based on sip signaling and midcom.
it would be worth to have a discussion on the following issue: 
  * path-coupled signaling is very benefitial if you have complex topologies
where routing symmetry causes packets towards different destination
addresses to hit different firewalls/nats. melinda shore raised this issue
in midcom (i guess for the first time). you can find her draft at
http://www.watersprings.org/pub/id/draft-shore-friendly-midcom-01.txt.
  * will people deploy firewalls in the future? (particularly since people
tunnel everything through http)? (my personal opinion is yes, they will.)

- nat issues: why do you think that nat handling is trickier? in fact the
case of a data receiver being behind a nat is (based on our investigations)
rather a nice case since you are assured that the data packets traverse this
nat (if you publish the obtained ip address, transport protocol + port as
your contact address). with a data receiver behind a firewall (in case of
complex topologies and routing asymmetry) things are more complicated
(without making additional assumptions). you mention some ipv6 specific
solutions that would fix the problem automatically. could you please
elaborate?

- you mention a few issues with regard to qos and charging. this is a
complex issue which we (engineers) are not really good in making a decision.
i have read a number of papers by prof. odlyzko (see for example
http://www.dtc.umn.edu/~odlyzko/doc/pricing.architecture.pdf). these papers
are nice to read but it is hard to estimate how individual issues raised in
these documents influence our work (and even in which time frame).

ciao
hannes

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Pekka Savola <[email-masked]>
Date: Thu, 26 Aug 2004 09:13:25 +0300 (EEST)

Thanks for your reasonable note, Hannes.

On Wed, 25 Aug 2004, Tschofenig Hannes wrote:
> - there have been a number of discussions about the bandwidth broker
> concept in the past and the disadvantages are known. (my personal
> opinion) if it boils down to the protocol details then the
> difference between the rsvp architecture with a pep/pdp and a
> bandwidth broker is not hudge. the central entity which performs the
> authorization (or policy) decision was already mentioned.

BB concept applied in which scope?  BB model, as with intserv model,
does not work (even well) across administrative boundaries.  Those
disadvantages are well known. Were those disadvantages related to
that, or something else?

> think such a decision would increase the number of usage scenarios.

The point is, if you just signal to a "broker" in the network you're
visiting (discovered in the same manner, e.g., as the recursive DNS
servers), it will know if you send it source, destination pair whether
the traffic goes beyond its control or not, and that avoids
unnecessary signalling (and signalling attempts).  Avoiding the 
unnecessary attempts is actually IMHO pretty important for multiple 
reasons.

> path-coupled signaling protocols in order to provide path-decoupled

It's interesting to note that there has been a BoF, but I'd be
interested as to why folks spoke about reusing the path-coupled
signalling (of course, it can be used for the lack of a better
protocol); too bad the BoF didn't submit any slides or minutes.
 
> http://www.welzl.at/research/projects/ip-options/).

Processing is indeed slower .. 10% or 30% is significant.  But what
I'm really worried about is that IP router alert -like options are
options which a hardware implementation cannot process.  An attacker
can just specify an undefined router alert option which forces the
processing to go to the slow path (instead of ASICs etc., it must be
put on the CPU), and overload the processing of the router that way.  
That's not good.  That can naturally be mitigated by rate-limiting the
IP options to some small value, but such would effectively disable the
processing of options, and require that you're able to match on those
options.  In other words, anything which has to be forwarded that
requires CPU processing seems bad to me..

Btw, you might be interested that in PMTUD WG session at IETF60 where
Michael Welzl proposed using IP options for PMTUD, Mark Allman (AFAIR)  
reminded of someone (him?) conducting tests on how big percentage of
hosts respond or not when there are IP options.  Michael's
presentation
(http://www.ietf.org/proceedings/04aug/slides/pmtud-4.pdf) already
said that 30% were ignored you completely (not just the IP option!),
but I think when conducting tests against web servers, this was even
smaller.  I don't know whether this is caused by broken firewalls or
what, but that certainly seemed to be showstopper at least for IP
options -based PMTUD.  Doesn't the same apply to NSIS if you'd like to
do it end-to-end?

> address denial of service issues at any place, regardless of the specific

IMHO the risks are very different.  With IP router alert -based
mechanisms, you need to address the DoS risks in all the routers in
the world.  With BB like centralized models, you just need to address
then in BB-like boxes: that's probably around 1%-5% of the number of
boxes where you need to do that, and that box is dedicated to just 
processing on protocol, so it should be easier to build 
protocol-specific anti-DoS measures.

> do it today with more complex mechanisms based on sip signaling and midcom.

Certainly, the operator can make a policy decision who can do it.  
And many protocols do create holes.  But the protocols don't ask the
firewall administrator whether they're allowed to do it or not; the
methods just use tricks (e.g., open an outbound session, and tunnel
traffic inbound through it) to pass the firewall whether that's
intended or not, right?

There could be some applicability to allow certain users, by policy,
to create pinholes (in such a manner that the administrator could
override it) in the firewall in the pre-determined small port range
(e.g., the hosts would still not be able to request pinholes for
TCP/80, but only for e.g., certain kind of SIP usage), but for
something more general than that.. I doubt it.

> http://www.watersprings.org/pub/id/draft-shore-friendly-midcom-01.txt.

I think you could actually address at least 95% of this if you just
supply (source,destination pair) to someone who has the copy of the
(IGP) routing data and can calculate what the path would be like.  
The rest 5% would be caused by policy-based custom routing (typically
only applies at the edge, so not an issue) and hosts with multiple
interfaces which send packets out through wrong interface with
incorrect source addresses.

> tunnel everything through http)? (my personal opinion is yes, they will.)

I also think yes, though I'm still hoping that some breakthroughs in 
host-based firewalls w/ centralized policy process would obviate the 
need for getting smarter edge firewalls.

> (without making additional assumptions).

There are a number of NATs which behave different ways, as described
in draft-jennings-midcom-stun-results-01.txt.  With a firewall, you
just need to enter one pass rule to enable a service.  With NAT state,
you might need one for each peer (port) that must traverse inbound
through it, unless you would be inserting some kind of "forwarder"  
state.

> you mention some ipv6 specific

I meant solutions like Teredo (draft-huitema-v6ops-teredo-02.txt) and 
similar other configured tunneling schemes which traverse NATs which 
will be able to provide non-NAT'ed IPv6 access, obviating the need for 
NAT traversal.

> these documents influence our work (and even in which time frame).

I totally agree that this is a complex issue, but I remain amazed that
there are still people who want to try to solve a complex issue
irregardless of its complexity even if there is history which seems to
indicate that it's likely that it isn't going anywhere.  Instead, I
would have been more happier if I had noticed that the folks would be
working on more easier tasks of the complex problem in a piecemeal
manner (e.g., the issues with qos and limiting at the first hop, or at
the predetermined constrained hops at the local site).

Trying to solve a broad problem might lead the problem being solved
with such means that the solution is never going to be widely deployed
to meet its original goals, and the effect is that we don't have any
realistic solution in the end. If a slightly different approach would
cause wider penetration on the really crucial issues (e.g., the issues
in the first hop, etc.), then we would possibly be much better off in
general.

That's my concern: (IMHO) it'd be better to concentrate on the easier
issues *only*.  End-to-end (without the constrains on where the ends
are)  signalling is not something that can be solved in a manner that
it would actually be used without reinventing the Internet
architecture.  Concentrating on the easier issues only would be
better, and not seeking (again, after RSVP :) the holy grail of
telecom/research QoS people, end-to-end signalling for reservations.
There's also smaller amount of economical/operational aspects to worry
about, as there are clearer areas of applicability and use for 1st
hop/predetermined constrained hops case.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Simon Leinen <[email-masked]>
Date: Thu, 26 Aug 2004 14:12:48 +0200

Hannes Tschofenig writes:
[...]
> - denial of service issues with the router alert option: it is true
> that router alert option processing is slower than packets which do
> not have the router alert option set. however, i am not sure whether
> this is truely a big issue based on the most recent information i
> saw (see http://www.welzl.at/research/projects/ip-options/).

This study compares the delay experienced by packets with IP options
to the delay experienced by packets without IP options, for a wide
variety of Internet paths.  It observes that on average, IP options
introduce between 7% and 26% (for different sets of paths at different
times) of additional delay.  The article says that "In any case, the
additional delay is clearly an order of magnitude smaller than the
RTT."

To some people, this may suggest that option processing overhead isn't
a problem in today's Internet.  I don't buy this at all.

The problem is this: A large part of the measured baseline delay is
due to propagation times (length of links/speed of light in fiber) and
thus mostly independent of whether IP Options are used or not.

If, as I suspect, the part of the delay that is actually due to
per-hop processing is small (for the baseline no-options case), then a
7%-26% increase of the *total* delay could mean an orders-of-magnitude
increase in per-hop processing overhead when IP options are used.

To illustrate this, here are the results of Michael's "extended ping"
test over a cross-town path with five GbE hops, sorted by RTT:

NONE   64 bytes from 130.59.35.42: seq=10007, ttl=62, rtt=0.171 ms
NONE   64 bytes from 130.59.35.42: seq=10009, ttl=62, rtt=0.172 ms
NONE   64 bytes from 130.59.35.42: seq=10005, ttl=62, rtt=0.178 ms
NONE   64 bytes from 130.59.35.42: seq=10001, ttl=62, rtt=0.202 ms
NONE   64 bytes from 130.59.35.42: seq=10000, ttl=62, rtt=0.209 ms
NONE   64 bytes from 130.59.35.42: seq=10008, ttl=62, rtt=0.211 ms
NONE   64 bytes from 130.59.35.42: seq=10003, ttl=62, rtt=0.216 ms
NONE   64 bytes from 130.59.35.42: seq=10006, ttl=62, rtt=0.220 ms
NONE   64 bytes from 130.59.35.42: seq=10004, ttl=62, rtt=0.225 ms
NONE   64 bytes from 130.59.35.42: seq=10002, ttl=62, rtt=0.233 ms
NOP    64 bytes from 130.59.35.42: seq=20005, ttl=62, rtt=0.735 ms
NOP    64 bytes from 130.59.35.42: seq=20003, ttl=62, rtt=0.769 ms
NOP    64 bytes from 130.59.35.42: seq=20006, ttl=62, rtt=0.775 ms
NOP    64 bytes from 130.59.35.42: seq=20000, ttl=62, rtt=0.803 ms
NOP    64 bytes from 130.59.35.42: seq=20008, ttl=62, rtt=0.803 ms
NOP    64 bytes from 130.59.35.42: seq=20004, ttl=62, rtt=0.816 ms
NOP    64 bytes from 130.59.35.42: seq=20002, ttl=62, rtt=0.885 ms
NOP    64 bytes from 130.59.35.42: seq=20009, ttl=62, rtt=1.749 ms
NOP    64 bytes from 130.59.35.42: seq=20001, ttl=62, rtt=3.456 ms
NOP    64 bytes from 130.59.35.42: seq=20007, ttl=62, rtt=4.288 ms

The optionless packets have an RTT between 171 and 233 microseconds
(other traffic on the path may explain the variance).  The packets
with NOP IP options have an RTT between 735 and 4288 microseconds.
This strongly suggests that IP options cause slow-path processing on
our routers.

Most operators won't worry about the additional delay incurred by
packets that use IP options.  But they WILL worry about the total
processing overhead in their routers when there are many such packets,
because slow-path resources are usually shared with the control plane,
and operators just hate it when control of the network is impacted by
user traffic.

I can see a few possible outcomes when IP options become more common:

1.) Routers implement option processing in the fast path
    - this will be expensive and/or take time if the fast path is "in
    hardware", e.g. mostly based on ASIC/FPGA/CAMs.  I think it will
    be hard to motivate this.
2.) Router vendors implement rate-limits for packets with
    (router-alerting) IP Options.  The rate-limiting must be done in
    the fast path, while options processing can remain in the slow
    path.  Maybe the rate-limits can be differentiated by
    option/extension header type, maybe not.
3.) Operators configure their routers to ignore IP Options.
4.) Operators configure their routers to drop packets with IP Options.

Regards,
-- 
Simon.

---

From: Michael Welzl <[email-masked]>
Date: 26 Aug 2004 14:40:11 +0200

Hi folks,

I would just like to add that our studies are ongoing, and

> additional delay is clearly an order of magnitude smaller than the

is only a part of our findings, which are only a part of
all the things that *should* be looked at. As Simon suggests,
processing overhead in routers may be an issue if they are
flooded with packets carrying options. Moreover, we also

> 3.) Operators configure their routers to ignore IP Options.

and this

> 4.) Operators configure their routers to drop packets with IP Options.

is already happening quite a bit nowadays.

Our findings are also limited to NOP options vs. no option
at all, and ICMP packets, not TCP. For different (even
more disappointing) findings with TCP and a new option
number, see pages 9/10 of:
http://www.icir.org/mallman/papers/tcp-evo-submit.ps

... who knows what's happening to packets that carry RA.

I believe that options are a very reasonable (and certainly
the most flexible) means of communication between end systems
and routers by design, and they could theoretically be used
for lots of great things in the future. The point of my
ongoing measurement effort is to show that they might not
be absolutely useless in the current Internet, and give
folks some concrete numbers - I don't believe that options
are ALWAYS useless (but don't ask me for a
counterexample  :(  ); depending on the envisioned usage
scenario, the measured properties will make things look
good or bad.

Cheers,
Michael

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 26 Aug 2004 23:47:50 +0200

On 26-aug-04, at 8:13, Pekka Savola wrote:

> But what

Pekka, please don't worry about this. There are very many things that 
can happen that require that the CPU look at packets: the hop limit may 
reach zero, the packet may be addressed to one of the addresses of the 
router, the packet may be too big for the next hop link...

Router vendors are able to cope with this with rate limiting. Really.

> (http://www.ietf.org/proceedings/04aug/slides/pmtud-4.pdf)

Ok, now this is a VERY stupid idea. I really don't get how such a large 
collection of smart people can come up with some many stupid ideas. 
This isn't going to work any better than current PMTUD works.

The very obvious only correct way to do PMTUD is for the correspondent 
to notice incoming fragments and inform the source of the size of the 
biggest fragments.

But that wasn't what we were talking about.

> already said that 30% were ignored you completely (not just the IP 
> option!),

It doesn't say, but I'm pretty sure this is IPv4, where options are 
long dead because they are just too bizarre and hardly used.

---

From: Pekka Savola <[email-masked]>
Date: Fri, 27 Aug 2004 09:06:24 +0300 (EEST)

> Router vendors are able to cope with this with rate limiting. Really.

All of these, except "packet may be addressed to one of the addresses
of the router" (obviously) are already situations which can be handled
(and AFAIK, are already handled in some equipment) on hardware.  
That's good.  The situation where the packet is aimed at one of the
addresses is also OK because you can actually set up filters for such
packets whether at the router or at your borders: it might not be
possible to check the "magic" packets which we're discussing.

Obviously, a specific kind of router alert option could be checked in
hardware as well, if the semantics were simple enough, there was
suffiicient customer demand, etc. (which there apparently isn't).  
But the real problem is that out of 16 bits of Router Alert option
codes, only about 3 values have been specified.  What does the router
do with an unspecified option code?  Even if it wanted, it couldn't
implement processing for the rest of the option codes, which are
unspecified, because it doesn't know how.  So either they need to be
ignored (which would lead to long introduction curve, if it would
require replacing hardware), or would need to be put on the CPU.  But
the latter provides an attack vector.