Subject: [RRG] Process proposal: agenda admission control

From: Tony Li <[email-masked]>
Date: Wed, 12 Dec 2007 13:16:32 -0800

Hi all,

As we all agree, RRG meeting time is our most limited resource, and  
perhaps also the most effective means to facilitate our convergence  
process. Thus, we would like to allocate adequate amount of time for  
discussions in our future meetings.  Consequently, we need to be more  
selective on the presentations of new proposals.  We would like to  
reiterate that the floor remains open for new proposals, so new  
proposals may still be submitted as a draft and considered on the  
mailing list.  However, we would like to ensure that only high  
quality proposals with substantive novelty will take group-wide time.

To assist us in this, the chairs plan to form a "program committee".   
These would be interested members who are respected architectural  
contributors.  For a proposal to be considered for an agenda slot,  
the backers of the proposal should approach members of this committee  
and ask them to review their proposal.  Committee members that have a  
conflict of interest (e.g., same employer) should recuse themselves.   
If any member of the committee feels that the presentation is  
worthwhile, it will be placed on the agenda, time permitting.  The  
identity of the endorsing member will be made public.

We would deeply appreciate your comments on this proposal.

Regards,
Lixia & Tony

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: David Meyer <[email-masked]>
Date: Wed, 12 Dec 2007 16:26:44 -0800

[Please excuse any duplicates; email snafu over here]

 	Tony,
 
> As we all agree, RRG meeting time is our most limited resource, and perhaps 
> also the most effective means to facilitate our convergence process. Thus, 
> we would like to allocate adequate amount of time for discussions in our 
> future meetings.  Consequently, we need to be more selective on the 
> presentations of new proposals.  We would like to reiterate that the floor 
> remains open for new proposals, so new proposals may still be submitted as 
> a draft and considered on the mailing list.  However, we would like to 
> ensure that only high quality proposals with substantive novelty will take 
> group-wide time.

	This makes good sense. 

> To assist us in this, the chairs plan to form a "program committee".  These 
> would be interested members who are respected architectural contributors.  
> For a proposal to be considered for an agenda slot, the backers of the 
> proposal should approach members of this committee and ask them to review 
> their proposal.  Committee members that have a conflict of interest (e.g., 
> same employer) should recuse themselves.  If any member of the committee 
> feels that the presentation is worthwhile, it will be placed on the agenda, 
> time permitting.  The identity of the endorsing member will be made public.
>
	Seems a reasonable proposal. However, the devil is in the
	details (as always). One point that seems particularly
	challenging is that "if any member feels that the
	presentation..., time permitting."  Its the "time
	permitting" part that is of concern. In particular, if we
	have N proposals and M < N slots (ie., time for M talks,
	not N), and all N proposals have proponents, how do you
	envision that being resolved?
	
	Dave

----- End forwarded message -----

---

From: "Hui Deng" <[email-masked]>
Date: Thu, 13 Dec 2007 09:00:33 +0800

I support this suggestion.
Neutral background would be necessary for one eligible for this committee.

-Hui

---

From: Tony Li <[email-masked]>
Date: Wed, 12 Dec 2007 17:09:42 -0800

> 	envision that being resolved?

Hi Dave,

I can see a number of different approaches, depending on differing  
absolute and relative values of M and N.  Here's some possible things  
that might happen:

a) Try to decrease N by elimination.  This could involve more reviews  
of the N proposals to see which ones the program committee feels are  
most interesting.  This might succeed if M and N differ only slightly.

b) Try to decrease N by rescheduling.  Some proposals might have to  
wait until the next meeting.

c) Try to increase M by adding meeting time.  Add more time slots at  
IETF, before or after IETF, or an outright interim meeting.

d) Something that we haven't thought of yet.

Obviously, this is something of a judgement call that we will have to  
make when we have both M and N in hand and can see the presentations.

Just to reiterate, please note that this process applies to *new*  
proposals.

Regards,
Tony

---

From: Tony Li <[email-masked]>
Date: Wed, 12 Dec 2007 17:21:02 -0800

> Neutral background would be necessary for one eligible for this  

Please note that we specifically did not require this.  It's very  
difficult to ascertain if someone is truly neutral and also ensure  
that they actually behave that way even if they say that they are.   
To avoid undue bias, we specifically want the supporting committee  
member to publicly known for supporting the proposal.  If the  
proposal is indeed unworthy, this would reflect badly on the  
committee member.  Hopefully this would act as a deterrent to  
insufficient selectivity or conflict of interest.

Tony

---

From: dimitri papadimitriou <[email-masked]>
Date: Thu, 13 Dec 2007 04:39:30 +0100

hi,

just curious, you state:

"only high quality proposals with substantive novelty will take 
group-wide time."

-> novelty against ? the existing global Internet, the existing 
scientific literature, the existing rrg proposals, etc. ?

"The identity of the endorsing member will be made public."

-> is the review itself also going to be made public/sent on the list ?

"If any member of the committee feels that the presentation is 
worthwhile, it will be placed on the agenda, time permitting."

-> meaning that one member agreement is sufficient (i guess so because 
you are referring to singular afterwards)

note: if a proposal comes that tries to improve an existing proposal are 
these also considered as "new" proposals ?

thanks,
-d.

Tony Li wrote:
> mailing list.  However, we would like to ensure that only high quality 
> proposals with substantive novelty will take group-wide time.
> 
> contributors.  For a proposal to be considered for an agenda slot, the 
> backers of the proposal should approach members of this committee and 
> ask them to review their proposal.  Committee members that have a 
> conflict of interest (e.g., same employer) should recuse themselves.  If 
> any member of the committee feels that the presentation is worthwhile, 
> it will be placed on the agenda, time permitting.  The identity of the 
> endorsing member will be made public.
> 
> .

---

From: Tony Li <[email-masked]>
Date: Wed, 12 Dec 2007 20:57:09 -0800

Hi Dimitri,

> scientific literature, the existing rrg proposals, etc. ?

Against the existing proposals.

> -> is the review itself also going to be made public/sent on the  

The review is essentially a yes/no answer and we'll only be  
disclosing the yeses.

> -> meaning that one member agreement is sufficient (i guess so  
> because you are referring to singular afterwards)

Correct.

> note: if a proposal comes that tries to improve an existing  
> proposal are these also considered as "new" proposals ?

Yes, we would want to treat a variant of an existing proposal as a  
new proposal.

Tony

---

From: "Hui Deng" <[email-masked]>
Date: Thu, 13 Dec 2007 13:37:27 +0800

Ok, I agree that publicly known for supporting the proposal could be a way
to guarantee the reputation.

-Hui

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 13 Dec 2007 11:47:36 +0100

Tony, Dimitri,

I think that the principle of selecting the presentations is necessary
and having a "committee" to decide is better than simply letting the
chairs decide on their own.

However, I'm surprised that you do not consider discussion on the
mailing list and submissions of drafts/technical reports.

> Against the existing proposals.

Assessing this novelty could be done by discussing publically the
submitted presentations on the mailing list before the meeting.

> The review is essentially a yes/no answer and we'll only be disclosing
> the yeses.

I don't think that a binary review is the proper way to select
presentations. In all scientific conferences, the selection on the
papers that will be presented is done based on the *text* comments
written by the reviewers. The feedback from the reviewers is more
important than the presentation slot.

If you want to have good proposals submitted to RRG for presentations,
may I consider the following :

- select a short committee whose job will be to review the proposals and
select those that will be presented given the available time
- choose a deadline for the submission of proposals, e.g. two months
before the meeting
- As I think that it is important for the community to be aware of the
proposals, they should be sent as internet-drafts or publically
available technical reports and announced on the mailing list
- during two or three weeks, the members of the committee write reviews
for the proposals. The community could also discuss the proposals
through the mailing list
- the committee reads all reviews and selects the best ones for
presentations. The reviews are sent on the mailing list
- the authors of the presentations should submit a revised version of
their proposal based on the feedback received on the mailing list and
from the reviewers

Olivier

-- 
http://inl.info.ucl.ac.be , Universite catholique de Louvain, Belgium

---

From: Scott Brim <[email-masked]>
Date: Thu, 13 Dec 2007 07:57:38 -0500

Excerpts from Tony Li on Wed, Dec 12, 2007 05:09:42PM -0800:
> a) Try to decrease N by elimination.  This could involve more
> reviews of the N proposals to see which ones the program committee
> feels are most interesting.  This might succeed if M and N differ
> only slightly.

Excerpts from Olivier Bonaventure on Thu, Dec 13, 2007 11:47:36AM
+0100:
> - select a short committee whose job will be to review the proposals
>   and select those that will be presented given the available time 
> 
> - during two or three weeks, the members of the committee write
>   reviews for the proposals. The community could also discuss the
>   proposals through the mailing list
> 
>   their proposal based on the feedback received on the mailing list
>   and from the reviewers 

I think public discussion of proposals should be the major factor
going into how much discussion they get in face-to-face meetings.
Individuals on the program committee should not be expected to or need
to make decisions on their own.  Therefore the process should include
a period of significant public technical review before a committee
member makes a decision.  "Significant public technical review" could
be as little as 2 weeks (more is better but sometimes there just isn't
time).  In general a good proposal gets *better* as it is scrutinized
and discussed.  In any case, once substantial public discussion has
taken place, then a contributor can ask the program committee to
consider the proposal.  This both protects the program committee from
suspicion of arbitrariness and technical incompetence :-p, and also
supports them in evaluating proposals.

---

From: "Fleischman, Eric" <[email-masked]>
Date: Fri, 14 Dec 2007 11:18:36 -0800

I'm sure that I'm not the only person who is uncomfortable with the
"fluff" in this proposed process.

I propose that the process be enhanced with the following modifications
that should occur well before any meeting or any identification of what
proposal is "worthy" or "unworthy":  each proposed approach (in order to
become eligible for consideration should)
1) create a terse summary of its technical approach (i.e., technologies
used with a goal for the RRG community to classify and contrast similar
and dissimilar approaches)
2) identify its preferred *deployment architecture" approach (i.e., to
identify where in the Internet "cloud" this approach operates and how it
interacts with other Internet entities)
3) It's advocates should identify what they perceive the approach's
technical strengths to be
4) It's advocates should identify what they perceive the approach's
technical weaknesses to be. 
5) It's detractors (if any) should compile a list providing technical
evidence to question the claimed technical strengths and add to the
perceived weaknesses

-----Original Message-----
From: Tony Li [mailto:[email-masked]] 
>Please note that we specifically did not require this.  It's 
>very difficult to ascertain if someone is truly neutral and 
>also ensure that they actually behave that way even if they 
>say that they are. To avoid undue bias, we specifically want 
>the supporting committee member to publicly known for 
>supporting the proposal.  If the proposal is indeed unworthy, 
>this would reflect badly on the committee member.  Hopefully 
>this would act as a deterrent to insufficient selectivity or 
>conflict of interest.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 16 Dec 2007 15:20:15 +0100

> - choose a deadline for the submission of proposals, e.g. two months

That's unworkable if the RRG wants to meet at every IETF meeting,  
because that effectively cuts the time available for submission in  
half and means that at the meetings, we'll be discussing stale, two- 
month-old proposals.

However, submitting new drafts the week before the meeting and  
assuming that people will read them so they can be discussed doesn't  
work, either.

> proposals, they should be sent as internet-drafts or publicly

Agree.

> reviews

I think this mechanism is too heavy. Don't forget that until now,  
there were few, if any, limitations on who got to present.

Personally, I'd rather see the presentation time limited rather than  
the number of new proposals that's presented. If you can't say it in  
10 minutes, you're trying to say too much anyway and you may as well  
have a reading break so people can read the draft.

And then maybe accept questions and remarks only from people who have  
read the draft in a first round at the microphones.

It might also be useful to have presentations and discussions on  
subsets of proposals rather than cover proposals in their entirety.

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 17 Dec 2007 08:37:38 +0100

Iljitsch,
> because that effectively cuts the time available for submission in half
> and means that at the meetings, we'll be discussing stale, two-month-old
> proposals.

No, it means that -00 drafts will be submitted two months before the
meeting and that the RRG will discussed revised drafts that have been
revised based on the received feedback. The drafts presented will be
more mature.

Olivier

---

From: Christian Vogt <[email-masked]>
Date: Mon, 17 Dec 2007 12:43:42 +0200

Tony and all -

The institution of a program committee for setting up meeting agendas is
a good idea in my opinion.  It seems reasonable that this committee
should consider mailing list discussions as part of its decision making.
The transparency of this review process will be an advantage over the
typically anonymous review process at scientific conferences.

The review process should be made as efficient as possible for members
in the program committee and reviewers on the mailing list.  To enable
this, new proposals should include a one-page summary of their gist and
of how they address the goals in [draft-irtf-rrg-design-goals].

Furthermore, I agree with Iljitsch that 10-15 min should in general be
sufficient for presentations at RRG.  This plus 5 min for a discussion
would allow for twice as many presentations during RRG sessions.

Kind regards,
- Christian

---

From: Jari Arkko <[email-masked]>
Date: Mon, 17 Dec 2007 15:31:35 +0200

I think its a good idea to think about what proposals deserve meeting
time, particularly if there are many new ones. And it might not always
be clear whether all proposals are actually addressing the scalability
problem that we should be talking about.

I would suggest, however, that this is only a part of the problem. I'd
like to see more of the meeting time spent on *discussion* of the issues
rather than presentation of the proposals.

And in general, the problem is perhaps not so much in whether we give
time for some new proposals in the agenda, but rather in getting a
proposal that has been presented to an actually workable solution. I
suspect that's where 98% of the effort lies in :-)

Jari