Subject: [Icar] Input based on SIRS experience

From: "Wijnen, Bert (Bert)" <[email-masked]>
Date: Fri, 9 Jan 2004 15:44:02 +0100

Forwarded with permission.
But probably only usefull after a WG gets instantiated.

> -----Original Message-----
> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: vrijdag 9 januari 2004 11:40
> To: James Kempf
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Solutions] SIRS dismissed as a failure
>
>
> My opinion is that we didn't generate enough interaction with
> the ADs over SIRs, and part of this was due to a technical gap -
> there was no link between the I-D tracker and the reviews. So I
> think it is very important for the real review system that it is
> somehow linked to the I-D (and hence to the I-D tracker) via
> the ietf.org web site. So that if you have the name of an I-D,
> you can not only find its status from the tracker, but also
> its reviews. They then become a real evaluation tool.
>
>    Brian

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Dave Crocker <[email-masked]>
Date: Sat, 10 Jan 2004 09:08:08 -0800

Adding my own thoughts to the archive, since the topic came up, and with the 
hope it can added to the discussion:

	It was not much effort to get a core of reviewers signed up.  

This suggests a good core of community motivation to help.  We have not needed 
more, so we do not know whether an effort like this can attract _enough_ 
qualified reviewers.

	I am not sure how to interpret the low number of review requests.  

It is easy to say that it is due to lack of community awareness or interest or 
trust in the SIRs effort.  Certainly we have not done as much marketing as we 
might have wished. However this is view is confounded by the very slow metabolic 
rate of the IETF and the fact that the effort began at the beginning of a 
summer.  

No doubt there probably are other factors.  Cliches like 'tipping point' and 
'crossing the chasm' were created for just this issue. For example, it may be 
that it is more difficult to change existing working group efforts than it is to 
get integrated in with new(er) ones. Certainly we have not done any serious, 
sustained effort to get working groups to request reviews. 

On doing the administrative side of this effort, there are some things that I 
suspect are strategic, besides the relatively obvious fact that this really does 
take some ongoing, administrative effort, more than I had expected.  (But, then, 
I always underestimate these things.)

	It needs to be easy to have a record of the review process for a document and, 
	by derivation, an ability to see the whole set of review efforts.  There should 
	be a central point for tracking review efforts and a central point for 
	accessing reviews.

Besides the obvious, local and tactical benefit to the individual working group, 
this permits the community to get the ability to review the _set _of reviews and 
get a sense of the effort and result of doing reviews.  In other words, it has 
marketing value.  It makes it easy for others to see that there is review 
activity and it makes it easy for them to evaluate the nature and quality of 
those reviews.  This will make them more comfortable getting reviews for their 
own efforts and will help them decide on what reviewers to ask for. Whether this 
needs to be a permanent, formal archive, like RFCs, an ephemeral, formal 
archive, like I-Ds, or an unofficial, ephemeral archive like a WG mailing list, 
has not been resolved.  

My own thought is that formal publication (RFC and even I-D) is far too much 
overhead.  The only existing mechanism is the mailing list. Creating a new 
mechanism strikes me as too much effort, and added effort is a very large 
barrier to adoption of this process.

Opening up the IETF Tracker to working group chairs, and adding the review 
construct and/or allowing free-form entries would help this enormously.  
(Free-form means that the IETF does not have to formally adopt the thing being 
tracked.  This means reviews -- and other activities -- could be tracked much 
sooner than would be possible if we first have to get IETF-wide approval of the 
review process.

	I am not sure whether selection of reviewers, by requesters,  needs to be 
	different than we set up.  

A web page with a list of bios, and a mailing list for sending requests to is 
probably fine.

	The public discussion about reviewing shows that the community is still working 
	on clarifying what types of reviews are needed, when, and how to use them.  
 
	There is also the small matter of the SIRS reviews that have been done.  

Have they been they type that are needed?  Should they be done differently?  We 
have not tried to assess this. Certainly we cannot expect regular, widespread 
use of a review process until working groups know what to ask for, or at least 
what to expect they will get and how they should use the results.

d/
--
Dave Crocker <dcrocker-at-brandenburg-dot-com>
Brandenburg InternetWorking <http://brandenburg.com>

---

From: "Spencer Dawkins" <[email-masked]>
Date: Sun, 11 Jan 2004 21:52:25 -0600

Dave,

Thanks for providing this input. Have I ever thanked you and Brian
publically for taking a swing at SIRS? It was an omission. Thank you.

I have a couple of comments inline.

Spencer

----- Original Message ----- 
From: "Dave Crocker" <[email-masked]>
To: "Icar (E-mail)" <[email-masked]>
Sent: Saturday, January 10, 2004 11:08 AM
Subject: Re: [Icar] Input based on SIRS experience

> more, so we do not know whether an effort like this can attract
_enough_

Yeah, Margaret is also coming up with 200-300 reviewers needed, and we
still have no idea how many we can get. :-{

> rate of the IETF and the fact that the effort began at the beginning
of a

I concur completely with each point.

> sustained effort to get working groups to request reviews.

Yes.

> be a central point for tracking review efforts and a central point
for

I agree, and reviews should be tightly tied to documents (easy to see
what reviewers think).

> barrier to adoption of this process.

This may be correct, but it's tragic, if so. Most of our archives
aren't that tractable, and some are REALLY grim (whether because of
spam or simply because of posting volume).

> sooner than would be possible if we first have to get IETF-wide
approval of the

I guessed that by "free-form" you were thinking of  using the
mechanism to track reviews of documents that hadn't been adopted as WG
documents yet. That's also a question.

> A web page with a list of bios, and a mailing list for sending
requests to is

The mailing list violates a concept Ted Hardie talked about in
Minneapolis - that appeals to individuals work better than appeals to
groups in the IETF. I think Ted is right. Requesters will really have
to be bothered to wonder who needs to review a document, rather than
posting a request and wishing reviewers would materialize.

> what to expect they will get and how they should use the results.

You are such a social scientist. How inconvenient that you should ask
a question when we have no idea what the answer is!

---

From: Dave Crocker <[email-masked]>
Date: Sun, 11 Jan 2004 20:26:37 -0800

Spencer,

>  Yeah, Margaret is also coming up with 200-300 reviewers needed, and 
>  we  still have no idea how many we can get. :-{

The calculations along those lines all look reasonable.  At the moment, 
I'd class it as a problem we'd love to have.  More realistically, yes, 
we need to make sure we design a mechanism that is both useful and 
scalable.

One thought is that good early reviewing might substantially alter what 
is needed later.

Another is to note a separate line of discussion that essentially 
distinguishes between some working group efforts that get more 
attention, versus others that might get less.

> >  My own thought is that formal publication (RFC and even I-D) is far
> > too much  overhead.  The only existing mechanism is the mailing 
> >  list.  Creating  a new
> >  mechanism strikes me as too much effort, and added effort is a very
> >  large  barrier to adoption of this process.
>  
>  aren't that tractable, and some are REALLY grim 

I think that long-term access to useful copies of all IETF working 
materials -- other than the stuff we formally publish -- has major 
benefit, for legal and academic purposes.  But I view this as a distinct 
issue.  My current concern is that the review stuff not create a new 
distribution and/or storage mechanism.

> >  Opening up the IETF Tracker to working group chairs, and adding the
> >  review construct and/or allowing free-form entries would help this
> >  enormously.  (Free-form means that the IETF does not have to 
> >  formally adopt the thing being
> >  tracked.  This means reviews -- and other activities -- could be
> >  tracked much sooner than would be possible if we first have to get 
> >  IETF-wide approval of the review process.
>  
>  mechanism to track reviews of documents that hadn't been adopted as 
>  WG documents yet. That's also a question.

It isn't the status of the document that I was thinking about.  It was 
the status of the _type_ of data being entered, in terms of IETF 
formality and officialness.  

If we simply let a working group chair put in any entry they want, the 
tracker won't know what the state transitions need to be, but the 
working group chair can set the state manually.  This means they do not 
need to wait for the IETF to define and approve the details beforehand.

> >  I am not sure whether selection of reviewers, by requesters,  needs
> > to be  different than we set up.
> >  
> >  A web page with a list of bios, and a mailing list for sending
> > requests to is  probably fine.
>  

The phenomenon of dissipated responsibility was first noted with the 
murder of Kitty Genovese, in New York, with lots of bystanders who could 
see each other, none of whom called the police.

That said, the queries that have come into the SIRS list have gotten 
good response.  However my point was not that all requests should come 
to the list, but that the list and the biographies with email addresses 
ought to be sufficient.

> >  There is also the small matter of the SIRS reviews that have been
> >  done...
>  

The purpose of an experiment is to spawn more experiments...

d/

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 12 Jan 2004 11:09:04 -0700 (MST)

On Sun, 11 Jan 2004, Dave Crocker wrote:

> the list and the biographies with email addresses ought to be
> sufficient.

In my experience as a SIRS user (though I was not able to solicit a
review), I found these two items highly desirable if not essential:

	- reviewer status (available immediately, busy with N
	  reviews, come back in 3 months, on vacation, etc.)

	- browseable collection of reviewer's past reviews

SIRs interface currently lacks them.

Thanks,

Alex.