Subject: Re: [Ieprep] alternate model for edge QoS

From: RJ Atkinson <[email-masked]>
Date: Sun, 1 Dec 2002 15:31:36 -0500

On Saturday, Nov 30, 2002, at 09:20 America/Montreal, Henning 
Schulzrinne wrote:
> To be more precise, we have to distinguish three types of states:
>
> (1) scheduling state; this only matters if you have per-flow queueing
> (2) filtering/policing state, which polices flows;
> (3) reservation state, one for each reservation, but only affecting 
> the control processor
>
> From what I can tell, (3) is not a major issue. Given the hundreds of 
> MB on a backbone router and the tens of MB on an edge router and the 
> roughly 500 bytes/reservation, you'd need lots of reservations to even 
> come close to mattering.

	Nope.  Each flow reservation is also likely to add complexity to (1) 
for
packet forwarding of the preferred flows and to (2) because the 
reservation
is probably internally a kind of filter.

	The German academic reports are not by themselves credible or 
sufficient.
More interesting and applicable reports would involve a typical 
mid-range
commercial IP router running a general deployment software load with a
typical packet forwarding load, typical sized routing table, and so 
forth.

> Web servers routinely handle large volumes of TLS connections.

	Web servers these days usually have *expensive* dedicated hardware
to handle TLS and in MANY cases the TLS has been off-loaded to an
*expensive* front-end processor (this box often is also the 
load-balancer
widget in front of the set of servers implementing the web site).

	Commercial routers do not have such dedicated crypto hardware at hand.
It could be added, but would significantly increase complexity of the
router and the user cost of the router -- and what about all the 
deployed
routers that won't be swapped out overnight.

>> Please do explain more.  I don't understand the details of who gives
>> whom a CERT about what, when, and when whom needs to do what with it.
>
> For concreteness, let's assume that FEMA is in charge of this 
> hypothetical program. FEMA sends, in some secure fashion, a cert to 
> each operator it cares to (based on whatever business relationships it 
> might set up). Each operator installs the CA cert in the control 
> processor, just like any other CA root cert.

	Right now, zero CA root certs are in any normally deployed control 
processor,
so you're assuming a bunch of stuff not in any mainstream deployed 
router today.

> Obviously, this happens during non-emergency conditions and can be as 
> simple as a signed S/MIME mail. There's no need for secrecy, just 
> integrity, since this is a public key only. Alternatively, the 
> operator can just download the .p12 file from the FEMA web server. 
> There may even be automated ways of doing this; I haven't followed all 
> the recent PKI-related WGs. This operation is repeated at cert 
> lifetime intervals; these seem to be roughly yearly.

	So the ISP now has to manage certificates in routers, make sure the 
certificate
stays in the router even if the CP gets swapped out due to a fault 
(faults of that
sort happened to me many a time on reasonably modern backbone routers). 
  Increases
the operational complexity and operational costs pretty significantly.

> When a signed resource request comes along that claims to be signed by 
> the FEMA CA, the control processor verifies this assertion. If this 
> checks out, the holder of the cert (presumably, an authorized 
> emergency worker) gets her reservation, if not, the attempt fails. No 
> real-time interaction with outside entities is required.

	It is an expensive computation to verify a signature.  An obvious DDOS 
attack is
to flood the router with forged resource requests and force the router 
to spend
lots of time performing a bunch of expensive computations.  Bad idea 
for one to enable
such router validation if one wants the router to keep BGP in sync, 
forward packets,
and do sundry other stuff needed to keep the Internet up and running.

	Sorry, this just is not deployable stuff right now, even if the fantasy
crypto accelerator hardware were shipping today in most commercial IP 
routers.

Ran

_______________________________________________
Ieprep mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ieprep

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 01 Dec 2002 16:28:00 -0500

> packet forwarding of the preferred flows

The whole point is that all of them are mapped (after filtering) to one 
class, so there is no additional scheduling state - it's one priority queue.

I just looked, more or less randomly, at the Juniper ERX edge router. It 
claims the ability to support 200,000 filters.

>  and to (2) because the reservation
> *expensive* front-end processor (this box often is also the load-balancer

The numbers I cited did not use any accelerator (the paper also contains 
numbers for those, which are indeed much higher).

> so you're assuming a bunch of stuff not in any mainstream deployed

Please recall that the discussion started with my assumption of adding 
NSIS functionality, which is likely to require both IPsec and possibly 
TLS, depending on how the design shapes up. Clearly, this is not 
something that you can plug into today's router. The point of the 
discussion was to see what, if any, requirements might be helpful for 
protocols being developed elsewhere in the IETF.

> the operational complexity and operational costs pretty significantly.

Again, the worse case scenario is that the cert is gone when the 
earthquake hits. Means you get no-preference service. I suspect many 
routers already support ssh, so they already deal with PK management, so 
this is hardly earth-shattering complexity.

>     It is an expensive computation to verify a signature.  An obvious
> DDOS attack is
> lots of time performing a bunch of expensive computations.  Bad idea for
> one to enable

As noted, low-end PCs can do hundreds of such verifications a second. 
This is more than enough for the purposes of what's needed here.

---

From: RJ Atkinson <[email-masked]>
Date: Sun, 1 Dec 2002 19:47:44 -0500

On Sunday, Dec 1, 2002, at 16:28 America/Montreal, Henning Schulzrinne 
wrote:
> I just looked, more or less randomly, at the Juniper ERX edge router. 
> It claims the ability to support 200,000 filters.

Probably it does, for the set of filters that are pre-defined in their
ASIC.  That doesn't say anything about the other state that's needed.
And off-hand it doesn't say whether or not any particular vendor's 
existing
ASIC filters can do the QoS filtering that has been discussed here.
No doubt some do and some don't.

Ironically, common layer-3 switches (from multiple vendors) do in fact 
have
an internal flow-cache architecture that lends itself to offering QoS
with less pain from state.  Common backbone routers (from multiple 
vendors)
use a very different internal longest-prefix-match architecture that has
more pain when one has to deal with flow state.  The LPM approach scales
better than a flow-cache approach when inside a backbone router.  In a
typical layer-3 switch application (LAN/MAN, edge, access), that 
dimension
of scalability is not as much of an issue.

> numbers for those, which are indeed much higher).

And most uses of TLS aren't related to what you are proposing
to make routers perform.

>> processor, so you're assuming a bunch of stuff not in any mainstream 
>> deployed
>> router today.
>

I think a very practical requirement is that the new proposal NOT 
create easily
exploited DOS or DDOS attack vectors.  So far, the proposals being 
discussed
all create such new attack vectors.

> routers already support ssh, so they already deal with PK management, 
> so this is hardly earth-shattering complexity.

No, the worst case is that the router falls over and dies because it 
gets DDOS
attacked -- or the network availability declines visibly due to the 
additional
complexity -- or other consequent operational problems make the network 
brittle
for a myriad of reasons.

Oh, and SSH doesn't really do certificate management today.  Also, the 
operators
apply filters to any deployed SSH (generally these drop all SSH packets 
that don't
claim to originate on the NOC subnet or from a small set of authorised 
hosts;
and filters throughout that domain's network protect against spoofing 
that subnet).
That protection/risk-management approach would not work for any 
customer-initiated
preferred service requests, because the emergency could happen anywhere 
and one
can't know in advance where the IEprep users will happen to jack in.

> This is more than enough for the purposes of what's needed here.

An attacker can trivially throw thousands of false packets at a router 
per second,
each of which forces the router to perform the signature verification, 
so your
math does not support the idea that it can trivially be done in a 
mainstream
CPU.  Instead, you are creating a new (and easily exploited) 
Distributed DOS
attack vulnerability for ISPs.  My guess is that they won't decide to 
deploy
that (and roll said as much in Atlanta).

Of course, inside a single private network a bunch of things might be 
useful
that aren't inside a multi-domain network.  My comments are focused on 
the
issues around inter-domain QoS, not on deployment inside a large private
network.  DoD's own IP networks (e.g. NIPRnet) are examples of large 
private
networks, of course.  So this is an area where what DoD IP networks do
and what GETS-like IP networks do will likely vary.

Ran

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 01 Dec 2002 22:40:46 -0500

On DDOS, it's a bit more subtle than throwing packets at a router: TLS 
(and others) make sure that only attackers with non-forged IP addresses 
even get checked. That way, you'd have to compromise hundreds of 
distinct hosts to even get one second worth of DOS.

RJ Atkinson wrote:

> And most uses of TLS aren't related to what you are proposing

The crypto operations are similar, so the performance is likely to be in 
the same ballpark.

> I think a very practical requirement is that the new proposal NOT create 
> easily
> CPU.  Instead, you are creating a new (and easily exploited) Distributed 
> DOS
> that aren't inside a multi-domain network.  My comments are focused on the

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 2 Dec 2002 10:18:05 -0500

On Sunday, Dec 1, 2002, at 22:40 America/Montreal, Henning Schulzrinne 
wrote:
> (and others) make sure that only attackers with non-forged IP 
> addresses even get checked. That way, you'd have to compromise 
> hundreds of distinct hosts to even get one second worth of DOS.

	Today, it is not possible to put in source address filtering on the 
largest
pipes inter-connecting two (or more) ISPs.  A lot of deployed router 
hardware can't
handle it.  Even if the router hardware can, it is intractably 
complicated to create
and correctly maintain (in real-time) the right source address prefix 
filter set.
Consider the issues faced by someone trying to source address filter an 
OC-192
connecting to one of the top 3 providers, each of which has meaningful 
amounts
of daily change in the set of prefixes that is legitimately connected 
to it.  This
is one of several reasons why the inter-domain QoS situation is 
fundamentally
different from intra-domain QoS.  Intra-domain QoS is not uncommon 
these days
using existing mechanisms (e.g. DiffServ, 802.1p, and so forth).

	Since there is no way to source address filter all inbound links,
there is no way for a backbone router to sort out the non-forged IP 
addresses
from the forgeries.

	And at @Home, our experience was that a single province in Canada (or 
any
large US state where we operated or any Australian state that we 
operated in)
typically had thousands of compromised hosts at any point in time.  
Those compromises
were not something that @Home (or any other ISP) could prevent.  Most 
compromises
were due to the home user not applying OS patches (that were in fact 
already available)
from Microsoft.

	Therefore I have no problem in considering thousands of zombie hosts 
being
available across the country/globe to launch a coordinated DDOS attack 
on an ISP.
It has happened numerous times in the past in the US and overseas.  
There is no reason
to believe that it won't happen in future (or is less likely in 
future).  @Home applied
(and the better of the cable network operators apply) source address 
filtering at the edge
so that IP address forgeries are minimised (not zero, but minimised), 
but lack of IP
address forgeries doesn't help the victim when a large scale DDOS 
attack is in progress.

> The crypto operations are similar, so the performance is likely to be 
> in the same ballpark.

	I dispute that the crypto operations are so similar.  Most TLS 
operations
are using symmetric ciphers (e.g. DES, AES), which are computationally
less expensive than for asymmetric ciphers (e.g. RSA).  Only asymmetric 
ciphers
are normally used for signing/verifying.  One could imagine a 
symmetric-only
situation using a Needham-Schroeder KDC (e.g. Kerberos V) to reduce the 
computational
burden, but that would still require a substantial deployment effort of 
its own.

	Adding complexity to any network tends to lead to reduced availability,
increased probability of a fault occurring (because more things can go 
wrong),
and longer mean-time-to-repair (MTTR).  It is important that an 
operator understand
all these tradeoffs before they deploy anything that increases the 
complexity
of their systems and network.  So far, the operators on this list are 
concluding
that the proposed inter-domain QoS would add so much complexity that it 
would
cause much more harm than good.

Ran
[email-masked]