Subject: RE: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics atOther Layers (pmol)]

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 1 Nov 2007 13:48:22 +0100

> -----Original Message-----
> From: Sam Hartman [mailto:[email-masked]] 
> Sent: Monday, October 29, 2007 10:24 PM
> To: Leslie Daigle
> Cc: IESG; [email-masked]; [email-masked]
> Subject: [PMOL] Re: A question about [Fwd: WG Review: 
> Performance Metrics atOther Layers (pmol)]
> 
> >>>>> "Leslie" == Leslie Daigle <[email-masked]> writes:
> 
> I doubt I'll use the output in security protocols.
> 

Isn't it true that best security protocol designs always take
performance aspects into account, because users will turn off the
security features if they deem their performance reduction too great? So
measuring the performance of security protocols and the impact of
activating security on other protocols and applications seems to be
important. 

Dan

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Dan Schutzer" <[email-masked]>
Date: Thu, 1 Nov 2007 09:22:54 -0400

agreed

-----Original Message-----
From: Romascanu, Dan (Dan) [mailto:[email-masked]] 
Sent: Thursday, November 01, 2007 8:48 AM
To: Sam Hartman; Leslie Daigle
Cc: [email-masked]; IESG; [email-masked]
Subject: RE: [PMOL] Re: A question about [Fwd: WG Review: PerformanceMetrics
atOther Layers (pmol)]

Dan

---

From: Sam Hartman <[email-masked]>
Date: Thu, 01 Nov 2007 11:09:58 -0400

>>>>> "Romascanu," == Romascanu, Dan (Dan) <[email-masked]> writes:

    >> -----Original Message----- From: Sam Hartman
    >> [mailto:[email-masked]] Sent: Monday, October 29, 2007 10:24
    >> PM To: Leslie Daigle Cc: IESG; [email-masked]; [email-masked]

    Romascanu,> Isn't it true that best security protocol designs
    Romascanu,> always take performance aspects into account, because
    Romascanu,> users will turn off the security features if they deem
    Romascanu,> their performance reduction too great? 

In many cases the performance of security protocols is not a huge issue at all with modern hardware.
There are a few important exceptions.

So measuring
    Romascanu,> the performance of security protocols and the impact
    Romascanu,> of activating security on other protocols and
    Romascanu,> applications seems to be important.

It has not been my experience that it is important to a level where
metrics are requested or used.  Certainly performance is something we
think about, and it is something that does get measured from time to
time and there are certain people who care about security performance
a lot.

Leslie asked for comments from uninvolved parties and I'm giving my
personal opinion that I would not find this work useful.  If others
do, we should go charter it.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 02 Nov 2007 09:40:38 +1300

> do, we should go charter it.

I think it will be useful, if it succeeds in rigorously defining
metrics for upper layer protocols, given their dependencies on
the metrics for lower layer protocols. The context is how to
write meaningful service level agreements for application level
services, which require well-defined and reproducible metrics.

    Brian

---

From: David R Oran <[email-masked]>
Date: Thu, 1 Nov 2007 17:25:41 -0400

> services, which require well-defined and reproducible metrics.
For the services I work on in my day job (e.g. multicast/unicast real- 
time video streaming) the metrics I am concerned with are either:

a) RTP/RTCP transport metrics, for which AVT has done a very good job  
with and hence PMOL is redundant, or
b) session protocol setup metrics (e.g. RTSP), which are mostly useful  
for debugging as opposed to service measurement
c) application-level SLA metrics, which are pretty abstract and not  
amenable to direct computation (e.g. visible artifacts per hour), or
d) encoding/perceptual metrics, such as PSNR, R-factor, etc. which are  
in generally defined on bodies other than the IETF and which have been  
the subject of much controversy both in the industry and in the IETF  
when people have tried to get them blessed.

As such, PMOL would be valuable to me if:
1. it succeeded in developing useful algorithms/methods for  
correlating these metrics to lower level metrics like route  
convergence time or congestive loss rates, or
2. it succeeded in developing application metrics which are less  
susceptible to industry controversy, vendor wars, service-provider  
"count everything that moves" behavior, and found wide adoption.

My personal assessment is:

a) the likelihood of PMOL succeeding at either of these is quite low,  
but I'm willing to be pleasantly surprised, and
b) if PMOL starts encroaching on the work in AVT I will be unhappy as  
I think AVT does a very good job without needing a new venue for  
developing useful transport level metrics, at least for real time flows

In summary, my general level of interest in PMOL is low, but that's a  
moot point since the WG has been chartered and I wish the group the  
best of luck!

Dave Oran (IAB hat off).

> PMOL mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/pmol

---

From: Bill Sommerfeld <[email-masked]>
Date: Fri, 02 Nov 2007 12:30:43 -0400

> issue at all with modern hardware.

Depending on the scope of this effort I see a bunch of things that might
be worth modelling:

 - additional compute resources (cpu, memory, crypto hardware, battery
energy, etc.)
 - additional round trips (that pesky speed of light thing)
 - increases in message sizes/reduced effective MTU
 - availability (if your security protocol depends on a KDC/OCSP
server/CRL distribution point/other service, if it's not available,
you're not available)

The first is not a factor on typical client computers, but the same is
not necessarily true for the mobile phone/pda class of widget or even
for servers -- in the latter case customers say they want high enough
utilization that the overhead of security protocols is going to be
significant for server sizing.

> metrics are requested or used.  

it's very common for customers to ask "how big a server/server farm do
we need to support this expected workload?".  The impact of security
protocols on that workload can be significant.  

					- Bill

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 2 Nov 2007 10:52:01 -0800

Hi -

> From: "Bill Sommerfeld" <[email-masked]>
> To: "Sam Hartman" <[email-masked]>
> Cc: "IESG" <[email-masked]>; "Romascanu, Dan (Dan)" <[email-masked]>; <[email-masked]>; <[email-masked]>; "Leslie Daigle"
<[email-masked]>
> Sent: Friday, November 02, 2007 8:30 AM
> Subject: Re: [PMOL] Re: A question about [Fwd: WG Review: PerformanceMetrics atOther Layers (pmol)]
...
> Depending on the scope of this effort I see a bunch of things that might
...

What's the relationship of all this to RFC 2564 and RFC 3729?
(For the phone case, the question would extend to RFC 4711.)

Randy

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2007 06:51:42 -0800

> There are a few important exceptions.

A significant one is the lack of hardware to support security protocols.
GigE interfaces have been common on both desktops and laptops for 5
years, but no corresponding security hardware has been similarly deployed.

The software performance of security protocols has been the more
substantial issue, and is likely to continue to be for the forseeable
future.

Joe

---

From: Stephen Kent <[email-masked]>
Date: Wed, 14 Nov 2007 15:39:50 -0500

Joe,

I disagree with your suggestion "The software performance of security 
protocols has been the more substantial issue, and is likely to 
continue to be for the forseeable future."

I suspect that most desktop users do not need hardware crypto for 
performance.  Irarely if ever drive my GiGE interface at its line 
rate. With fast processors, especially multi-core processors, we have 
enough cycles to do symmetric crypto at data rates consistent with 
most application demands for individual users.  Public key operations 
for key management are usually low duty cycle, so they too can be 
accommodated.

Steve

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2007 13:35:53 -0800

Hi, Steve,

Stephen Kent wrote:
> protocols has been the more substantial issue, and is likely to continue
> to be for the forseeable future."
> 
> performance.  Irarely if ever drive my GiGE interface at its line rate.

It's not hard to drive it high enough to see a substantial impact
(300+Mbps); when I turn on S/W crypto, that drops to less than 1/3 at
best. See the paper below.

> With fast processors, especially multi-core processors, we have enough
> cycles to do symmetric crypto at data rates consistent with most
> application demands for individual users.  Public key operations for key
> management are usually low duty cycle, so they too can be accommodated.

Public key is less the issue. See the following for recent measurements
using multicore processors - FWIW, this will peg the processing of a
modern CPU just to reach over 100Mbps:

J. Touch, Y. Yang, "Reducing the Impact of DoS Attacks on Endpoint IP
Security,"Proc. NPSec 2006, in conjunction with ICNP 2006, Nov. 2006.
http://www.isi.edu/touch/pubs/npsec2006

Joe

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 14 Nov 2007 22:32:58 +0000

On Wed, 14 Nov 2007 15:39:50 -0500

> for key management are usually low duty cycle, so they too can be
Thanks -- I was going to say something similar.  I regularly back up my
laptop's disk over a software-encrypted GigE link. The dump file
occupies about 35G of disk space; it takes about 70 minutes.  Exclusive
of protocol overhead, that comes to ~71M bps; given IP, TCP, and ssh,
I'd guess it's more like 75-80M bps.  I also know that I can run ttcp
between that pair of machines at about 500M bps.  Am I really suffering
from a 7:1 performance hit from the crypto?  Nope.

I can't do controlled experiments right now, since the laptop and
server in question are currently about 350 km apart at the moment.

Dumping the disk to /dev/null took about 30 minutes.  The maximum hit
I'm taking from the crypto, then, is about 2.3:1.  Using the
performance numbers from 'openssl speed' for 1024-byte blocks for
AES-128 and SHA-1, the crypto is 23 minutes of CPU time.  (This is a
1.8Ghz, single-core laptop.)  The other 17 minutes probably go to data
copies -- I'm doing 'dump | ssh cat >file' -- so a kernel implementation
(say, IPsec) would be better -- and overhead, plus (of course)
calculation error).  The point, though, is that the crypto isn't the
limiting factor.  It's not stopping me from doing anything.  It's not
even the biggest bottleneck in my application.  And *that's* what's
important -- not *network* speed, but *application* speed.

		--Steve Bellovin, http://www.cs.columbia.edu/~smb

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2007 16:25:55 -0800

Hi, Steve,

> from a 7:1 performance hit from the crypto?  Nope.

By essentially shutting your machine down for over an hour.

> I'm taking from the crypto, then, is about 2.3:1.

The numbers I get are 3:1 for some protocols, worse for others, but the
CPU hit is also substantial, vs. just taking time but not CPU to do the
transfer.

> Using the

Your application shut your machine down for over an hour - just dumping
 alone wouldn't). That's substantial - sure, not for you, but you run
crypto. For those who don't, this is a large part of why (the other part
is key management, but that doesn't come into play for things like
backups since it's a single key and two known parties).

---

The point is that you have a GigE interface you don't use. Why? It's
cheap, and when you do use it - to dump at work - it's a lot faster than
100Mbps.

Crypto puts a huge dent into that capability the way it's done right
now, and nobody is selling me 1Gbps crypto chips for a price that makes
it cost-effective to put it into a laptop. And I don't exactly want to
burn 100% of my CPU capacity to saturate my 802.11 link, let alone such
a small fraction of my GigE.

Joe

---

From: Sam Hartman <[email-masked]>
Date: Wed, 14 Nov 2007 21:22:48 -0500

>>>>> "Joe" == Joe Touch <[email-masked]> writes:

    Joe> Hi, Steve,
    Joe> Steven M. Bellovin wrote:
    >>> I disagree with your suggestion "The software performance of
    >>> security protocols has been the more substantial issue, and is
    >>> likely to continue to be for the forseeable future."
    >>> 
    >>> I suspect that most desktop users do not need hardware crypto
    >>> for performance.  Irarely if ever drive my GiGE interface at
    >>> its line rate. With fast processors, especially multi-core
    >>> processors, we have enough cycles to do symmetric crypto at
    >>> data rates consistent with most application demands for
    >>> individual users.  Public key operations for key management
    >>> are usually low duty cycle, so they too can be accommodated.
    >>> 
    >> Thanks -- I was going to say something similar.  I regularly
    >> back up my laptop's disk over a software-encrypted GigE
    >> link. The dump file occupies about 35G of disk space; it takes
    >> about 70 minutes.  Exclusive of protocol overhead, that comes
    >> to ~71M bps; given IP, TCP, and ssh, I'd guess it's more like
    >> 75-80M bps.  I also know that I can run ttcp between that pair
    >> of machines at about 500M bps.  Am I really suffering from a
    >> 7:1 performance hit from the crypto?  Nope.

    Joe> By essentially shutting your machine down for over an hour.

I'm only going to send this one message, but then I'll drop out of the
thread.  We've drifted far from Leslie's original query.

Steve did not say his machine was CPU bound.  Also, even if it was CPU
bound, he's probably running an operating system with reasonable
multiprogramming characteristics.  So, if he wanted to use the
machine, his backup would take longer, but he'd get to do whatever he
wanted.

Yes, Steve almost certanily did  slow down any heavy CPU use during the time when he was doing the backup.

Our point--Steve, Steve and I--is that for a lot of uses and a lot of
users, no one cares.

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 15 Nov 2007 02:47:18 +0000

On Wed, 14 Nov 2007 21:22:48 -0500

>     Joe> By essentially shutting your machine down for over an hour.

No, not at all; not even close.
> the time when he was doing the backup.
Right. Operating systems are really good at sharing the CPU between
processes.  As long as you're talking about pure CPU load, the
scheduler works *very* well, and has for O(45 years).  Right now, as
I'm typing this, I'm running that dump again in the background, this
type piped to 'cat >/dev/null' to better simulate the pipe overhead,
and I'm running 'openssl speed' in another window.  I do not perceive
any slowdown whatsoever.  I would notice it I tried to view a movie, or
if I tried to do serious picture or video editing, or if I tried
running another command -- say, firing up firefox if it were not
already running -- that did a lot of disk I/O.  I don't know of any OS
-- and that includes every version of Unix I've ever used, going back
more than 30 years, and every version of Windows I've used, going back
about a dozen years, that shares disk bandwidth particularly well.  But
sharing CPU is very easy.

I don't dispute Joe's numbers about how crypto can limit bandwidth.
Again, though, what matters to me is system and application
performance, and there I generally don't feel the hit.

Why do I use GigE?  Well, in one sense it's because that's what came
with my computer, but I did spend the money to convert my house network
to GigE.  There's some unencrypted traffic to the file server.  Just as
important, even with the crypto I'm approaching the limits of what I
can do with 100BaseT.  As I noted, my actual, over the wire, dump
operations are running at 75-80M bps.  There's not a lot of headroom
between that and the 91M bps that is the fastest I've managed with
ttcp.  I just got a new, faster laptop; I'm curious what it will do,
with and without the crypto.

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Nov 2007 22:43:01 -0800

Sam Hartman wrote:
...
> Our point--Steve, Steve and I--is that for a lot of uses and a lot of

Perhaps that's why everyone is using security. We don't have a problem then.

Joe

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 15 Nov 2007 13:53:23 +0000

On Wed, 14 Nov 2007 22:43:01 -0800
Joe Touch <[email-masked]> wrote:

> > Our point--Steve, Steve and I--is that for a lot of uses and a lot
> > of users, no one cares.
> 
I didn't say that; I said that performance generally isn't the issue.
Often, there's a *perception* of a performance issue, because once
there was. The bigger problem, in my opinion, is usability.  *Lots* of
people use SSL, because they don't have to do anything.  SSL as used in
https has lots of problems I won't go into here, but it is excellent
protection against passive eavesdroppers.

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Nov 2007 06:35:33 -0800

> protection against passive eavesdroppers.

While I'm sure your anecdotal laptop measurements are valid, there are
plenty of others who:
	- transfer large files over disks with more than 70Mbps of BW
		e.g., photos are now over 15MB/file, and videos larger
	- do enough with their CPU in the meantime that they would
	  notice when the OS was sharing it - e.g., photoshop

Why don't users turn on security on DSL lines? They do - VPNs, SSL, etc.
Sure, various protocols still have problems, as you note, but security
over low-speed links is largely a success story.

Why don't core Internet routers have security? Note that some router
vendors sell "IPv6" routers even though they don't come with IPsec HW,
and don't put IPsec in SW -- even though it begs the question of what
IPv6-IPsec should be called (IPv5.5?).

Why? They do more than ASCII email when they're trying to send packets,
have a SATA disk (even on laptops), and don't have - or want - CPU power
to burn (or even to share 50/50) to hide a very real performance problem.

Performance problems come in many flavors:
	- per packet overhead
	- algorithmic overhead
	- keying overhead
	- policy lookup overhead

All of these are real problems, and can cause performance to drop to a
small fraction of what's capable without security.

Hiding the problem with other debilitations hasn't made it go away.

Joe

---

From: Stephen Kent <[email-masked]>
Date: Thu, 15 Nov 2007 10:35:18 -0500

Joe,

This discussion  seems to have moved from a discussion of crypto use 
on home/office computers, to use in routers. There is no good 
motivation for other than edge (CPE?) routers to make use of IPsec 
for subscriber traffic.  We know, from discussions with operators, 
that use of IPsec to protect BGP is a non-starter, because of where 
in the router the processing would be done (given current router 
designs).  In any case, use of IPsec by routers is a very different 
topic that use in home/office computers and ought not be brought into 
this discussion.

As for the original topic, yes, performance hits come in various 
flavors when we discuss crypto protocol use. For example, there was a 
good paper at NDSS a few years ago that showed how "marshalling" of 
data in  SSL implementations was a very big part of the performance 
hit. Nonetheless, the bottom line is that for mainstream users, most 
of us are not convinced that performance is the primary reason for 
not using crypto.

Steve

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Nov 2007 11:23:20 -0800

Stephen Kent wrote:
> Joe,
> 
> This discussion  seems to have moved from a discussion of crypto use on
> home/office computers, to use in routers. There is no good motivation
> for other than edge (CPE?) routers to make use of IPsec for subscriber
> traffic.

BGP...

> use of IPsec to
> protect BGP is a non-starter, because of where in the router the
> processing would be done (given current router designs).

Yes - and that was the punchline that performance does matter.

> In any case,
> use of IPsec by routers is a very different topic that use in
> home/office computers and ought not be brought into this discussion.

They are two different things, agreed.

> As for the original topic, yes, performance hits come in various flavors
> when we discuss crypto protocol use. For example, there was a good paper
> at NDSS a few years ago that showed how "marshalling" of data in  SSL
> implementations was a very big part of the performance hit. Nonetheless,
> the bottom line is that for mainstream users, most of us are not
> convinced that performance is the primary reason for not using crypto.

If "us" means crypto folk, I agree.

If "us" means the rest of us - who don't use crypto - I am not at all
convinced. There are a variety of other communities who want to use
security - high performance (grid, optiputer), enterprise (huge numbers
of short connections), etc. They all have different reasons for not
using crypto more, but writing off performance would be to continue a
mistake.

I've made that point clear; whether it's actually heard or not isn't
something I have much control over, though.

Joe