Subject: Re: I-D ACTION:draft-etal-ietf-analysis-00.txt

From: "John Stracke" <[email-masked]>
Date: Thu, 28 Mar 2002 14:23:05 -0500

>there's nothing wrong with looking at aggregate data about working 
groups'
>production of RFCs.  of course, one should be careful about conclusions.

And the authors do caution that their numbers are blind to the quality of 
the RFCs.  Their point, though, is that looking at the easy metrics is 
better than not measuring anything at all; it gives a first-order 
approximation.  If someone then figures out how to refine that 
approximation, then great.

The authors don't seem to make any substantive conclusions based on this 
approximation.  Instead, they looked at what they learned in the process 
of gathering the data, and suggest improvements to WG charters to make it 
possible to gather better data in the future (making the charter 
machine-readable, and having it include a log of the events in the WG's 
lifetime).  Seems harmless to me.  (Possibly too much work--I can't speak 
to that--but harmless.  :-)

/==========================================================\
|John Stracke                    |Principal Engineer       |
[email-masked]   |Incentive Systems, Inc.  |
|http://www.incentivesystems.com |My opinions are my own.  |
|==========================================================|
|Some days, it just doesn't pay to gnaw through the straps.|
\==========================================================/

---

From: Lars Eggert <[email-masked]>
Date: Thu, 28 Mar 2002 12:12:21 -0800

John Stracke wrote:
> better than not measuring anything at all;

Wrong information is worse than no information. If the results don't 
mean anything, why measure?

Lars
-- 
Lars Eggert <[email-masked]>               Information Sciences Institute
http://www.isi.edu/larse/              University of Southern California

---

From: Mark Atwood <[email-masked]>
Date: 28 Mar 2002 12:25:12 -0800

"John Stracke" <[email-masked]> writes:
> approximation.

I disagree.

Some metrics (lines of code written per day, number of bugs found per
person, etc) are *actively* harmful to gather & report.

Counting RFCs looks like it's bad the same way that pure LOC counts
are bad.

Saying "we must measure *something*" is the Politician's Fallacy ("we
must do something, this is something, therefore we must do this.")

-- 
Mark Atwood   | Well done is better than well said.
[email-masked] | 
http://www.pobox.com/~mra

---

From: [email-masked]
Date: Thu, 28 Mar 2002 15:39:31 -0500

> mean anything, why measure?

Often, just knowing *why* the results dont mean anything is informative..

Back about a century ago, these two chaps named Michaelson and Morley tried
to measure the speed of the Earth through the ether.  Their results didn't
mean anything either, as Einstein managed to explain a few years later.

This is a -00.txt version.  It might make sense for the -01 version to
include text on why given metrics are/aren't percieved as good measures
by the IETF community....
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: Ian Cooper <[email-masked]>
Date: Thu, 28 Mar 2002 12:52:23 -0800

--On Thursday, March 28, 2002 12:25 -0800 Mark Atwood <[email-masked]> wrote:

>> And the authors do caution that their numbers are blind to the quality
>> of  the RFCs.  Their point, though, is that looking at the easy metrics
>> is  better than not measuring anything at all; it gives a first-order
>> approximation.
>
> I disagree.
>

True, though I thought LOC counting was done as an initial metric until 
(much) better things were found.

> must do something, this is something, therefore we must do this.")

I found the parts of the document that would enable more subjective 
measurements (like documenting the progress of documents within the group) 
more interesting than the actual "counting".

---

From: Marshall Rose <[email-masked]>
Date: Thu, 28 Mar 2002 13:36:02 -0800

> Counting RFCs looks like it's bad the same way that pure LOC counts

err, okay, i guess.

however, it may be useful for folks to actually read the draft before making comments... thus far, i've only seen two folks with comments who claim to have actually read the thing.

as an author of the draft, i don't think "Counting RFCs" was a goal, or even something we explicitly did...

/mtr

---

From: Matt Crawford <[email-masked]>
Date: Thu, 28 Mar 2002 16:31:50 -0600

>  however, it may be useful for folks to actually read the draft
> before making comments... thus far, i've only seen two folks with
> comments who claim to have actually read the thing.

OK, here's a new data point: I read it all and I have no comment.  It
is neither more nor less than it purports to be.  What's all the fuss?

---

From: Paul Robinson <[email-masked]>
Date: Fri, 29 Mar 2002 13:53:33 +0000

> (much) better things were found.

Actually it's what poor management did until (much) better management were
found to replace them. LOC has no value to anybody other than people who
want to fire you for replacing a 2000 line C++ program with one line of
shell script.
 
> more interesting than the actual "counting".

The question is, why measure? Management in large corps measure things
because they have nothing else to do and would rather look at a series of
pie charts than actually go and talk to guys on the shop floor. What is the
objective perceived in measuring anything around the IETF? If you read the
draft, pretty early on we encounter the following:

   "Historically, the IETF has an excellent track record; however, as it has 
   grown, there often is concern that IETF efforts are becoming less 
   efficient and perhaps less effective.  To date, the community has relied 
   on its subjective sense of this change.  Evaluation techniques are needed 
   that are both objective and useful."

At no point is there an explanation as to why these evaluation techniques
are required - to say there "is often concern" is a bit of a cop out. Now,
I'm not saying this shouldn't have been written and the authors have wasted
their time, but am I the only one who thinks this smells a little of an
attempt at the over-engineering of a voluntary group? Not a criticism, just
an observation, and I'd be grateful for input on why measuring performance
is something that should be applied to the IETF. Oh, actually, I think it's 
a great idea if we're going to link it all to performance related pay... :-)

-- 
Paul Robinson

---

From: Melinda Shore <[email-masked]>
Date: Fri, 29 Mar 2002 09:21:37 -0500

At 01:53 PM 3/29/02 +0000, Paul Robinson wrote:
>Now,
>attempt at the over-engineering of a voluntary group? 

Probably not, but I don't think so.  My bias is that more
information is always better than less information, and since 
a lot of people are unhappy with how things are going (working
groups that never finish, documents and charters taking a long,
long time to get through IESG review, etc.) finding ways to tickle
out possible indicators doesn't strike me as a bad thing.

Two more things: 1) I don't see any engineering going on in or
around the document - just the beginning of an analysis, and 2) 
voluntary groups ought to be held accountable to their participants, 
if not the consumers of their work.

Melinda