Subject: Should I* opinions be afforded a special status? (Re: [saag] Declining the ifare bof for Chicago)

From: Lakshminath Dondeti <[email-masked]>
Date: Tue, 12 Jun 2007 12:17:23 -0700

Folks,

If you want the history of this thread, please see the SAAG mailing list 
archive.

Thomas,

Your ideas that the "IETF is a meritocracy" and that "I* opinions are 
afforded special status" are to say the least worry me.  How do those, I 
wonder, fit with what's written in the IETF mission web page 
http://www.ietf.org/u/ietfchair/ietf-mission.html?

Your slides on "Bringing new work to the IETF" presented at the Prague 
meeting that I have just looked at today also seem to be in 
contradiction to the IETF mission.  Your idea that some people's 
opinions are afforded more weight than others' is certainly not how the 
consensus process works.  Do smarter people hum louder or get to raise 
both of their hands?  What are you saying?

We do not select our leaders for their technical knowledge alone.  See 
the oral tradition part of 3777.

The idea that somehow the ADs and the IAB are above the rest of the 
contributors is just wrong.  They are judges of consensus when 
appropriate and the consensus better be independently verifiable.  In 
the end, the entire process works with the IETF Community's consensus 
where the IAB and the IESG get to prioritize the work.

regards,
Lakshminath

On 6/12/2007 7:10 AM, Thomas Narten wrote:
>> I don't understand your line of reasoning for turning away this
>   work.
> 
> Actually, I thought Sam's note was quite clear. And I applaud his
> willingness to say no, if the effort isn't ready for another BOF.
> 
> Speaking as a former AD, it can be a very tough call to say yes/no to
> a BOF. Unfortunately, there is often interest, but interest is most
> definitely not enough. There needs to be more than interest. There
> needs to be a reasonable chance of a positive, forward-moving
> outcome. But in my experience, it is often the case with 2nd BOF
> requests that substantative issues have been raised already (on the
> list or in a previous BOF), but have not subsequently been adequately
> addressed by the BOF proponents. In such cases, another BOF will
> more-or-less just rehash what is already known, with no change in the
> overall status. It is the AD's job to judge whether that is likely to
> happen (in which case "no" is the right answer).
> 
> For a second BOF, a real danger with allowing it to go forward is that
> it raises false expectations on where things are heading.  There is a
> semi-written rule that says "no more than 2 BOFs". Thus, the second
> BOF needs to lead to a WG or an end of the effort. Once the second BOF
> has happened, there is (too often) an expectation that a WG will be
> formed.
> 
>> You note that there is a lot of interest, and I know that there has only 
>> been one person who disagrees with the IFARE proposal (perhaps I* 
>> members' opinions are afforded special status?).
> 
> Yes, I* opinions are afforded special status. They are our chosen
> leadership, and with leadership comes responsibility. Responsibility
> to be sure that if the work goes forward, it is well scoped, has a
> reasonable likelyhood of success, etc. And please remember, the IETF
> is a meritocracy. So please don't raise the "I* has special status"
> issue as if it were some kind of unfair or biased way of doing things.
> 
>> If you really want to help, at least now, please schedule the BoF and 
>> make suggestions on how to build support for the work (we have more than 
>> a month before the Chicago meeting).  Otherwise, it might be worthwhile 
>> to remove the IESG note in 4285.
> 
> I'd suggest you read (or reread) draft-narten-successful-bof-02.txt.
> 
> It is the BOF proponent's job to build support for an effort. If they
> cannot do so, that (for better or worse) says something about the
> importance/interest in the work. And note that "support" means not
> just finding those in favor of an effort, but getting those opposed to
> the effort (as currently scoped) to be satisfied by rescoping the
> effort or addressing their concerns (and again, there is a judgement
> here). I.e., there needs to be a sufficient degree of consensus on
> having the work go forward (with an agreed upon scope, etc.).
> 
> Thomas
> 

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Melinda Shore <[email-masked]>
Date: Tue, 12 Jun 2007 16:10:53 -0400

On 6/12/07 3:17 PM, "Lakshminath Dondeti" <[email-masked]> wrote:
> They are judges of consensus when

It seems to me in practice that as long as one group of people
is designated as having the authority to block work, approve
work, block document publication, approve document publication,
and so on, you don't have the kind of level playing field
you're describing.  I am aware that there are people (possibly
a lot of people) who feel that anything that's got "consensus"
behind it (for whatever value of consensus) should be considered
okay, but I don't recall us (institutional "us") ever having that
conversation and resolving that question.

Melinda

---

From: Sam Hartman <[email-masked]>
Date: Tue, 12 Jun 2007 16:29:36 -0400

>>>>> "Lakshminath" == Lakshminath Dondeti <[email-masked]> writes:

    Lakshminath> Folks, If you want the history of this thread, please
    Lakshminath> see the SAAG mailing list archive.

    Lakshminath> Thomas,

To be clear I'm not sure that I* opinions have been given special
treatment in this case.  So, carry on with the discussion, but be
aware that it may not actually have any binding to a specific incident
in the IETF.

---

From: Lakshminath Dondeti <[email-masked]>
Date: Tue, 12 Jun 2007 13:56:13 -0700

I don't disagree.  I do not claim that this is necessarily binding to 
any specific incident in the IETF.  I removed the ifare tag line from 
the subject.  I want the issue discussed at a much higher level than any 
one decision.  Individual decisions have state associated with them, and 
we don't need to discuss them with the community at large.

Lakshminath

---

From: Ted Hardie <[email-masked]>
Date: Tue, 12 Jun 2007 14:22:05 -0700

>>issue as if it were some kind of unfair or biased way of doing things.

Knowing Thomas pretty well, I think this is just an unfortunate juxtaposition
of two different statements.  The IETF, as a whole, values technical
input (rather than raw voting).  Statements which are backed by analysis,
careful thought, and reasoned arguments will be given more weight
than a +1 on a mailing list.   As 3935 states, the IETF is committed to
taking that input from anyone: "the issues on which the IETF produces its
documents are issues where the IETF has the competence needed to
speak to them, and that the IETF is willing to listen to technically competent
input from any source".  The weight assigned is from all concerned,
but it is up to the working group chair/ADs to assess the community's
reaction to a proposal or other technical input.  They assess, in some form,
how the community weights the technical arguments.  So the IETF
leadership is, in effect, given a responsibility related to that assessment.

That does not mean the IETF leadership is itself a meritocracy; it's not.
The IESG and IAB are picked by NomComs for a variety of skills and
"fit" is a critical one.  Someone who can fit into the team the NomCom is
building may be selected over someone who is equal or better in any or all
of our technical disciplines, because of the need to balance skills, personalities,
and time commitments.  This last is a particularly important point, as the
IETF is led, fundamentally, by people who have the will, time, and resources
to dedicate to that effort.  There will likely always be people who have more
"merit" on some objective scale than those who are selected, but who cannot
serve because they lack time or support. 

In assessing an IESG or IAB member's reaction to a document, BoF proposal,
I think we should consider the *time* they put into it; each  has dedicated
significant time to getting an overall context for the IETF, which may involve
knowledge of what the competing proposals for attention are and other
dragons in the upcoming road.  That may give weight to their opinion, but it
has to be matched by time spent on analysis of the specifics.  An IESG member
who puts a discuss on a document with "this feels wrong" is given considerable
pressure to make an actionable statement instead, with a real analysis backing it;
the same is or should be true for statements during a BoF or WG meeting.

For an AD to say: "I don't see community consensus on this, here's how you
could demonstrate it" is more than fair; it is actionable.  For an AD to say
"I don't see how this would get deployment, so I don't want to spend IETF
cycles on it, here's what it would take to demonstrate it could get deployed"
is also actionable.  But neither statement relies on a "meritocracy", they
rely on management.  For *anyone* to say "I don't think this is technically
correct or the optimal solution" will require the same analysis, careful thought,
and reasoned analysis that is the hallmark of good IETF input.  That includes
statements by members of the IESG or IAB.   It will also require follow-up,
and continued engagement on the technical solution; if it doesn't, then the community
needs a statement on why the IESG or IAB don't think the problem is worth
the time. In other words, if there won't be *technical* engagement, there needs
to be a clear management reason given for that.

BoFs have been hard in this space for a long time.  But since they are how the
IETF remains a technical vital, relevant place to do work, I think they are
worth the time it takes to respond to proponents, either technically or as a
management issue, on why they are or are not going forward.

			regards,
				Ted Hardie

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 12 Jun 2007 21:24:40 -0400

I may well be misreading Lakshminath below.
But the note as written seems to say that ADs are only supposed to 
judge consensus.
That misses important parts of the point.
They are also selected for technical judgement, and expected to use 
that judgement.
So, for example, an AD is NOT required to sponsor an individual 
submission, nor required to sponsor a BoF.
If they think the topic is a bad idea, or not in the IETFs area, or 
otherwise would not make a good WG< they are supposed to say no.
Heck, they are supposed to say no to some working group charter 
requests, even when their is interest.  (For example if they think 
that the proposed work plan will not lead to an effective outcome.)
To be specific, that judgement is to be used for a lot more than just 
prioritizing the work.

Now, they should say "no" clearly, with a good explanation of the 
reasons.  And they should be open to hearing responses which may 
change their view.  But that does not change the fact that they are 
expect to exercise judgement.

Yours,
Joel M. Halpern

At 03:17 PM 6/12/2007, Lakshminath Dondeti wrote:
>Folks,
>
>If you want the history of this thread, please see the SAAG mailing 
>list archive.
>
>Thomas,
>
>...
>
>We do not select our leaders for their technical knowledge 
>alone.  See the oral tradition part of 3777.
>
>appropriate and the consensus better be independently 
>verifiable.  In the end, the entire process works with the IETF 
>Community's consensus where the IAB and the IESG get to prioritize the work.
>
>regards,
>Lakshminath

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 13 Jun 2007 07:17:13 +0200

We trust the IESG not only to judge consensus but also to provide
technical mentoring and leadership. We trust the IAB not only to
arbitrate in case of disagreement about consensus but also to
provide architectural insight and leadership.

Yes, that gives them a special status. It's called 'primus inter pares'.

    Brian

---

From: Lakshminath Dondeti <[email-masked]>
Date: Tue, 12 Jun 2007 22:40:15 -0700

I understand the technical judgment argument, but I see a lot of 
practical issues with it.  First, an AD (or an IAB member) may not be an 
expert in all the topics under review; in fact it is probably unfair to 
assume that they are.  Some of them seek help from the community (hear 
both sides of an argument, which to me is the right way to do it) and 
others seem to rely on one of their colleagues' opinion (I have heard 
people say, X thinks this is wrong/right and I believe them more than I 
believe Y who I have never met).  That's only human, but it could also 
be wrong.  What if X were biased and Y has a valid argument, which may 
or may not be immediately apparent to all (add to that language and 
cultural barriers, we may have serious practical issues).  I can go on, 
but it sounds really bad, so we won't go there.

I appreciate your idea of checks and balances (your last paragraph), but 
I think they are not enough.  Some ADs don't follow process.  Even when 
explanation is sought on a topic, some ADs make light of the issue and 
others are non committal.  Pointing out an issue sometimes is considered 
a problem.  The recall process is a hammer and we never dare to actually 
use it.  I want to also accentuate the positive: there are also many 
ADs/IAB members who take extra steps in the interest of fairness and 
engage in a dialog when there are issues.  That is of course very much 
appreciated.

I am generally of the idea that concentration of power, that too without 
transparency and without practical checks and balances is dangerous.  I 
would have thought it is a universally held opinion (perhaps with the 
exception of people who hold such power).

regards,
Lakshminath

On 6/12/2007 6:24 PM, Joel M. Halpern wrote:
> But the note as written seems to say that ADs are only supposed to judge 
> consensus.
> They are also selected for technical judgement, and expected to use that 
> judgement.
> requests, even when their is interest.  (For example if they think that 
> the proposed work plan will not lead to an effective outcome.)
> reasons.  And they should be open to hearing responses which may change 
> their view.  But that does not change the fact that they are expect to 
> exercise judgement.
> 
> Yours,
> Joel M. Halpern
>

---

From: Lakshminath Dondeti <[email-masked]>
Date: Tue, 12 Jun 2007 22:57:31 -0700

My education did not include Latin, but Wikipedia says there are several 
kinds of "Firsts among equals."  One example is a 'president' and 
another is a 'chair of an organization.'  Surely the first is not inline 
with our famous saying

"We do not believe in kings, presidents, or voting.
We believe in rough consensus and running code,"

Perhaps that saying is no longer true.  As it is we vote, we believe in 
people with almost unchecked powers, and the consensus processes are 
ignored with impunity.

Let me stop being cynical and say that it would be worthwhile to iron 
out the review criteria as clearly as possible.  The now "dead" discuss 
criteria document is a good example of the right direction.

thanks,
Lakshminath

---

From: Jari Arkko <[email-masked]>
Date: Wed, 13 Jun 2007 13:59:45 +0300

Lakshminath,

Just commenting from my own experience with & views
about the BOF process...

I don't think the fact that someone is in the I* means
their opinions necessarily carries more weight than
other opinions.

This does not imply that all opinions carry the same
weight. Informed, well justified opinions are taken
more seriously. We listen more to people who are
involved in an issue or whose networks are affected
than outsiders. We care about management concerns,
e.g., completing existing efforts before starting new
ones, so we listen to the involved managers.

When considering a BOF request, I personally
try to think about what will happen at the BOF
and after it. One way of doing this is to contact
experts in the field, managers (chair, AD, external
SDO manager, etc) of related efforts, customers of
for the proposed functionality, etc. And listen to what
they say. This gives you a pretty good prediction
of what the mike line in the meeting would say,
and helps you determine whether the effort is
ready for a BOF. The readiness level is important,
particularly for a 2nd (last) BOF.

Now, some subset of the people listed above are
likely to be found in the IESG or IAB. As you probably
know, the IESG and IAB have a teleconference where
proposed BOFs are discussed, and we get both the
type of input falling in the above categories, as well
as generalist/architectural feedback. When there
is significant discussion about a proposal from some
members in this call or when I know someone is
a key expert on the topic, I try to get them involved
on the list, conference calls with the proponents, etc.
Because, clearly, there needs to be a discussion, but
also because we need to keep our discussions out
in the open. This is not always possible, but in those
cases the AD has to understand the issue well enough
to take his or hers own position, be able to explain
it to others, etc.

But at the end of the day, the AD has to make a
decision based on the input he or she has gotten
and own views. He needs to weight the different
inputs appropriately, understand if there are any
significant technical issues, etc. The own views are
important, too;  the ADs should be guiding the effort
towards an improvement in the Internet, not merely
observing opinions of others. In any case, the AD has to
justify the decision on his own, not by merely
referring to expert opinions.

Anyway, my experience is that I often find myself torn
between (a) granting a BOF request because which is what
the proponents want and (b) asking them to do something
else such as re-scope or write documents because that is
what I feel would take us to the desired end result sooner.
The desired end result is, of course, published specs and
a better Internet, not merely having a meeting.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Wed, 13 Jun 2007 14:11:41 +0300

Laksminath,

> out the review criteria as clearly as possible.

I would support a document giving clearer guidelines
for this. Draft-narten-successful-bof already
outlines some of the things that a BOF proposal
should satisfy. If you or someone else has new text that
goes beyond this, I'd be interested in taking a look.

> The now "dead" discuss criteria document is a good example of the
> right direction.

Yes -- but for clarification to others (I know you are aware of this), the
criteria are in active use by the IESG and often referred to when we
talk about some AD's discusses. The document was originally an
I-D (now expired), but is available from:

http://www.ietf.org/u/ietfchair/discuss-criteria.html

Jari

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Wed, 13 Jun 2007 05:54:32 -0700

Fit might be the right criteria if the objective here is to have a nice jolly time.

We have a rather serious responsibility here. Many of the best people in the field are not exactly known for being easy to get along with. 

> -----Original Message-----
> From: Ted Hardie [mailto:[email-masked]] 
> Sent: Tuesday, June 12, 2007 5:22 PM
> To: Lakshminath Dondeti; Thomas Narten
> Cc: IETF Discussion
> Subject: Re: Should I* opinions be afforded a special status? 
> (Re: [saag] Declining the ifare bof for Chicago)
> 
> >>
> Knowing Thomas pretty well, I think this is just an 
> unfortunate juxtaposition of two different statements.  The 
> IETF, as a whole, values technical input (rather than raw 
> voting).  Statements which are backed by analysis, careful 
> thought, and reasoned arguments will be given more weight
> taking that input from anyone: "the issues on which the IETF 
> produces its documents are issues where the IETF has the 
> competence needed to speak to them, and that the IETF is 
> willing to listen to technically competent input from any 
> source".  The weight assigned is from all concerned, but it 
> is up to the working group chair/ADs to assess the 
> community's reaction to a proposal or other technical input.  
> They assess, in some form, how the community weights the 
> technical arguments.  So the IETF leadership is, in effect, 
> given a responsibility related to that assessment.
> 
> The IESG and IAB are picked by NomComs for a variety of 
> skills and "fit" is a critical one.  Someone who can fit into 
> the team the NomCom is building may be selected over someone 
> who is equal or better in any or all of our technical 
> disciplines, because of the need to balance skills, 
> personalities, and time commitments.  This last is a 
> particularly important point, as the IETF is led, 
> fundamentally, by people who have the will, time, and 
> resources to dedicate to that effort.  There will likely 
> always be people who have more "merit" on some objective 
> scale than those who are selected, but who cannot serve 
> because they lack time or support. 
> 
> In assessing an IESG or IAB member's reaction to a document, 
> BoF proposal, I think we should consider the *time* they put 
> into it; each  has dedicated significant time to getting an 
> overall context for the IETF, which may involve knowledge of 
> what the competing proposals for attention are and other 
> dragons in the upcoming road.  That may give weight to their 
> opinion, but it has to be matched by time spent on analysis 
> of the specifics.  An IESG member who puts a discuss on a 
> document with "this feels wrong" is given considerable 
> pressure to make an actionable statement instead, with a real 
> analysis backing it; the same is or should be true for 
> statements during a BoF or WG meeting.
> 
> For an AD to say: "I don't see community consensus on this, 
> here's how you could demonstrate it" is more than fair; it is 
> actionable.  For an AD to say "I don't see how this would get 
> deployment, so I don't want to spend IETF cycles on it, 
> here's what it would take to demonstrate it could get deployed"
> is also actionable.  But neither statement relies on a 
> "meritocracy", they rely on management.  For *anyone* to say 
> "I don't think this is technically correct or the optimal 
> solution" will require the same analysis, careful thought, 
> and continued engagement on the technical solution; if it 
> doesn't, then the community needs a statement on why the IESG 
> or IAB don't think the problem is worth the time. In other 
> words, if there won't be *technical* engagement, there needs 
> BoFs have been hard in this space for a long time.  But since 
> they are how the IETF remains a technical vital, relevant 
> place to do work, I think they are worth the time it takes to 
> respond to proponents, either technically or as a management 
> issue, on why they are or are not going forward.
> 
> 			regards,
> 				Ted Hardie
>

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Jun 2007 08:18:50 -0700

>I-D (now expired), but is available from:

Isn't this what the ION series is for?

--Paul Hoffman, Director
--VPN Consortium

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 13 Jun 2007 19:46:06 +0200

> Isn't this what the ION series is for?

Sure, but someone has to do the work to IONize each such document.
That's why I built the ad hoc list still to be found at
http://www.ietf.org/u/ietfchair/opNotes.html

    Brian

---

From: John C Klensin <[email-masked]>
Date: Wed, 13 Jun 2007 20:49:21 -0400

--On Tuesday, June 12, 2007 14:22 -0700 Ted Hardie 
<[email-masked]> wrote:

>...
> That does not mean the IETF leadership is itself a
> meritocracy; it's not. The IESG and IAB are picked by NomComs
> for a variety of skills and "fit" is a critical one.  Someone
> who can fit into the team the NomCom is building may be
> selected over someone who is equal or better in any or all of
> our technical disciplines, because of the need to balance
> skills, personalities, and time commitments.  This last is a
> fundamentally, by people who have the will, time, and resources
> to dedicate to that effort.  There will likely always be
> people who have more "merit" on some objective scale than
> those who are selected, but who cannot serve because they lack
> time or support.
>...

Ted, while I agree with what you say above, it suggests two 
things to me:

(1) The  best way to get a good fit, and an IESG that can work 
well together, is to  be sure that the IESG is as homogeneous as 
possible, with all people having the same perspectives and 
opinions.   On the other hand, if one wants maximum opportunity 
for, e.g., alternate sponsors for BOFs and documents should a 
given AD turn one down and for checks and balances more 
generally, it would  be wise to have as much diversity on the 
IESG as possible... diversity that is not necessarily strongly 
correlated with harmony.  We can only hope that Nomcoms 
understand that tradeoff and balance it very carefully.

(2) There has been a tendency in recent years for the scope and 
workload of the IESG to increase.  Whether that trend is 
intentional or not, increases in IESG workload tend to reduce 
the number of people who will have the time, resources, and 
support to take on IESG positions.   If the community wants 
maximum diversity, or maximum "merit" in those positions, it is 
very much in the community's interest to find ways to bring the 
workload down or at least to resist proposals and changes whose 
consequence is to increase the workload.

best,
    john

---

From: Thomas Narten <[email-masked]>
Date: Thu, 14 Jun 2007 17:08:39 -0700

> afforded special status" are to say the least worry me.

If you start from a postion that one cannot trust the I*, or WG
chairs, etc. (as a number of your recent postings seem to do), then
yes, one can't help to be troubled. however, if your basic starting
point is one of distrust, it's hard to imagine rules or process or
some other management or oganizational structure that will actually
work and also prevent abuse. If there is bad intent, very few rules
will prevent bad actions.

IMO, you have to have a structure/process/rules that assumes people
are generally trying to do the Right Thing. For checks and balances,
you then also need appeals procedures and a willingness to speak up
and challenge parties when there is evidence of bad decision making.

In my experience, the vast majority of our leadership do try to do the
right thing. And when confronted with having made a bad decision,
there is usually a genuine effort to fix things and do the right
thing.

> How do those, I wonder, fit with what's written in the IETF mission
> web page http://www.ietf.org/u/ietfchair/ietf-mission.html?

> both of their hands?  What are you saying?

If a respected security expert (one who has reviewed many documents,
contributed significantly to WG efforts, etc.) comes to a WG and says
"there is a problem here", but 5 WG members stand up and say "I
disagree and don't see a problem", do you really expect the security
expert's opinion to be given strictly equal weight and to just be
overruled since 5 voices are greater than 1?

> contributors is just wrong.

They are not above the rest in the sense of having absolute power and
having to answer to no one. Anyone is free to (and should) challenge
their arguments and decisions when they don't appear to be sound.  And
clearly they have to be able to defend their positions and give
concrete or "actionable" justifications.  But to somehow think they
are just "equals" would relegate them to just doing process, and
taking away their ability to use judgement. I for one do not want to
go there because it will almost certainly lead to bad outcomes because
the process "rules" didn't allow for someone to just step and and say
"no".

Others have also said things I agree with (e.g., Ted and Jari
especially).

Thomas

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 15 Jun 2007 08:58:11 +0200

Lakshminath Dondeti wrote:
> afforded special status" are to say the least worry me.  How do those, 
> I wonder, fit with what's written in the IETF mission web page 
A procedural point, and an opinion of principle.....

The IETF mission, as approved by the IETF consensus process, is RFC 3935.

Offhand, I don't recognize the page you're pointing at, but Google's 
only link to it 
(<http://www1.ietf.org/mail-archive/web/ietf/current/msg30045.html>) is 
dated March 2004, so I suppose I'm to blame for its existence (current 
IETF chair: I very much recommend author, date and context also for 
"informal" statements - having sinned greatly against that in the past, 
I now see the error of my ways...)

But I believe that in neither that page nor in RFC 3935 did we ever 
commit the fallacy of saying "one man, one vote".
How the weight one gives to opinions is distributed varies, I believe - 
both from case to case and from person to person - but we've never 
committed the error of saying "all opinions are equal".

We are very committed to "all opinions should be HEARD" (we do demand 
civility, but that is about form, not content), but we've never claimed 
to weigh them equally.

HOW the weights are distributed is a far longer discussion.

                     Harald

---

From: Ted Hardie <[email-masked]>
Date: Fri, 15 Jun 2007 08:40:42 -0700

At 8:58 AM +0200 6/15/07, Harald Alvestrand wrote:
>But I believe that in neither that page nor in RFC 3935 did we ever commit the fallacy of saying "one man, one vote".
>How the weight one gives to opinions is distributed varies, I believe - both from case to case and from person to person - but we've never committed the error of saying "all opinions are equal".
>
>We are very committed to "all opinions should be HEARD" (we do demand civility, but that is about form, not content), but we've never claimed to weigh them equally.

We have espoused "rough consensus and running code" as a credo for a long
time, though, and Thomas's most recent formulation (where one well-known
contributor is weighted over the opinions of five not-known contributors) does
not fit that rubric, at least in my opinion.  It sounds like an organization where
the old-dogs network runs things and can exclude input from anyone whose
opinion doesn't agree with theirs.  Since one of the key values of the IETF
is openness, we have to be careful with giving that impression; as an open
organization where a newcomer is only free to agree with the old dogs isn't worth
much.

I think part of the part of the problem here is that we're talking about opinions.
In a real working group, the chairs can tell who is actually reading the documents,
producing text, and coding up implementations.  They can weight based
on *participation*, in other words, in the chartered work of the group.  In a BoF,
especially where some of the proposers are new participants or are referring
to needs outside the comfort zones of many of our participants, this is much
harder.  The tendency to weight participation in the *IETF* higher than
familiarity with the work means, at least if I'm reading the tea leaves right,
that we will be turning away new participants as well as new work.  Any one
of Thomas's five will likely look at the treatment received in that BoF and decide
to work elsewhere. 

I think BoF reform along the lines Bernard has suggested might help solve that,
by creating a phase where participation in the *work* can be judged much as
it is in a chartered working group.  After a "BoF period", the chairs/ADs/community
can better judge who really is familiar with the area, who is working to
achieve the proposed work, and who will likely implement and deploy it; those
all deserve real weight in our process.  That does not mean that those outside
the group cannot and should not be heard; intra-area review and cross-area
review are a key part of our value.  But it allows us to value our new contributors
as equals to our known contributors.
			regards
				Ted Hardie

---

From: Michael Thomas <[email-masked]>
Date: Fri, 15 Jun 2007 09:04:08 -0700

> overruled since 5 voices are greater than 1?
Context matters here. I've seen situations where the so-called "expert"
is unable to articulate the problem, is rocking a personal hobby
horse of theirs, is expressing a general philosophic point without much
attention to the actual problem (see "unable to articulate"). Etc. Also:
it may just be me, but this gradual creep toward "experts" having a
named, and different status is bothersome ("expert review"). It sweeps
under the rug that this is really a continuum of cluefulness, experience,
as well as frankly political considerations, some of which are more
odious than others.

       Mike

---

From: Lakshminath Dondeti <[email-masked]>
Date: Sun, 17 Jun 2007 22:43:14 -0700

Thanks for your response Thomas.

Apologies if I had inadvertently given the impression of mistrust in the 
current leadership, I* and WG chairs.  I have very good working 
relationships with most if not all of the I* folks I interact with. 
Sure, there have been differences of opinions, but with communication, 
the differences were either resolved or we agreed to disagree.  I try to 
communicate rather than hold things in and that may have given the 
impression you are getting.  Also, we seldom debate about all the things 
that went right. :)

In at least one of my emails I tried to balance things out by pointing 
out the positive.  I should have tried harder.

With that clarification, I will try and respond to some of your notes below:

> will prevent bad actions.

In most cases, I am simply seeking more transparency and determinism in 
our operation.  One of the things that has come about from the 
discussions is that perhaps a clear cut list of things to be done 
post-BoF-session from the AD and/or the IAB would be useful.  Now some 
ADs already do this and others are trying to get there.  That is a 
positive thing.

What this might remove is an appearance of arbitrary or biased decision 
making.  The I-D tracker is a great example of this.

If indeed there were abuse or improper bias, such processes might help 
independent evaluation.

> and challenge parties when there is evidence of bad decision making.

Agreed.  I guess I challenge when there is an appearance of bad decision 
making.  I would rather not wait until things get out of hand and then 
attempt use of the hammer of appeals.  That is just my personal preference.

> there is usually a genuine effort to fix things and do the right

Agreed, the vast majority are indeed trying to do the right thing and 
provide the right guidance based on their experiences in the leadership 
positions.  It is also important to point out that some of the positions 
are extremely time consuming.  I appreciate the I*'s and WG chairs' 
service to the IETF very much.  Many in fact go out of their way to 
reach out and communicate and that is also very much appreciated.

> overruled since 5 voices are greater than 1?

Sure, but the the notions of "respected" and "expert" are fuzzy.  I have 
used that argument myself in the past, but it gets us into trouble 
because the experts venture into topics where they are not experts at in 
any sense of that word.  Bernard and John allude to some of the related 
issues.

The notion of "expert opinion" is also trouble for another reason.  It 
is a longer story and somewhat indirect, so I won't go into that here.

> concrete or "actionable" justifications.  

With that clarification, I don't see us being that far off from each 
other in our understanding of the operation of the IETF.

I was challenging a decision and incorrectly thought that it was based 
on the opinion of one or two I* member(s).  I do believe that in the 
specific case, there is no clear cut technical issue for such an opinion 
to be weighted over others (I would say though that if there were a 
clear technical issue, it doesn't matter who made the opposing case. 
The technical issue should more or less stand on its own).

> But to somehow think they

I am not quite saying this either; our process allows ADs to take 
decisive actions, but there are limitations, and rightfully so.  Some of 
the limitations are not clear cut and so we may have more of these 
debates to get a clear sense.  In one of my emails I made the point that 
there are misunderstandings in some corners of the IETF that everything 
is based on consensus; not everything is and  that happens to be the 
consensus.

In closing, I learned some things and some good things have come about 
because I raised the issue.  Some of those things could have happened 
even due to private discussions.  But I hope others on the list found 
the open debate beneficial.

Thanks again for sharing your thoughts.

best regards,
Lakshminath (minus the dots, just in case there is any ambiguity)

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 18 Jun 2007 11:28:26 +0200

> as well as frankly political considerations, some of which are more

Mike, just to try clarify the "expert review" issue. That term comes
up specifically for IANA assignments that are controlled more than
First Come First Served and less than RFC Required. There's a full
discussion in draft-narten-iana-considerations-rfc2434bis but my
short version is: we can't expect IANA to have in-house expertise
for every registry, and we don't want an IETF debate about every
relatively routine assignment. Having designated experts seems to
be the only reasonable and scaleable solution.

I prefer personally to use a term like "specialist" for review
of drafts, or "generalist" in the case of Gen-ART reviews. You
can be a specialist without being an expert :-)

      Brian

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 18 Jun 2007 08:30:11 -0400

I think this actually highlights where I am concerned.

At 01:43 AM 6/18/2007, Lakshminath Dondeti wrote:
>In most cases, I am simply seeking more transparency and determinism 
>in our operation.

I agree that transparency is a good think.  (There are a few cases 
where that must be sacrificed, but very few, and there needs to be 
good reason.  I applaud the IESG having more detailed minutes, more 
example, as an improvement I thought was not practical, but works.)

On the other hand, determinism is what I get from a government 
bureaucracy.  Determinism means that the person is required to follow 
the rules, even whether their judgement says that this is a different 
case, and needs to be handled differently.  BoF formation and working 
group chartering are examples of places where I want intelligent 
judgement exercised.  As such, the result will not be 
"deterministic."  There should be no set of hoops you can jump 
through that guarantee a BoF, or a working group.

In fact, as has been observed in other (procedural) contexts, we 
usually do better when we can have fewer rules (quasi-absolute 
determinism) and better guidelines which give good understanding of 
the intent, and leave the leadership room to figure out how the 
intent is best achieved.

Yes, we should have a clear understanding of what the normal case is, 
and what one would expect to be the parameters for success or 
failure.  And yes, if the leadership decides not form a working group 
after a BoF or two, we should expect an understandable explanation. 
(BoF sponsorship is harder, but still refusal ought to be explained.)

Yours,
Joel M. Halpern

---

From: Michael Thomas <[email-masked]>
Date: Mon, 18 Jun 2007 09:50:51 -0700

> be the only reasonable and scaleable solution.

Yet, this not the only place where I've seen "expert review"...
> can be a specialist without being an expert :-)
which I take to be what you're referring to here. Let me say that
I'm happy to get cross functional review whenever I can get it. But
the way that Thomas posed his question "should 5 wg ignorants
overrule one `expert'?" shows that there can be a downside when,
in fact, the `expert' is off in the weeds for the reasons (and more)
I mentioned above. Since "expertry" is really a continuum of
competence and that becoming a recognized "expert" is one part
clue, one part clique, and probably one part availability, we shouldn't
give their expertise _too_ much power -- especially on well worn
ratholes. Also: as this "expert review" thingy becomes more popular --
and powerful -- parceling it out to a single person has a tendency to
pave over what happens when subject area experts meet: they often
disagree. Therein lies danger for the unhappy working group on
the receiving end: you're just postponing that fight for later in the
process. Worse is when the "expert" demands satiation of a favorite
hobby horse, typically of the kind that other experts find either
uninteresting, or not worth the fight since there's no skin off their
nose.

So getting back to Thomas' question: as always with engineering,
"it depends.", and I hope that those sending out the "experts" are
cognizant of that as well.

       Mike

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 18 Jun 2007 18:31:13 -0700

On Jun 12, 2007, at 12:17 PM, Lakshminath Dondeti wrote:

> appropriate and the consensus better be independently verifiable

I would be very interested in hearing your ideas on how the IESG  
should be doing steering or how the IAB should be be doing  
architecture. Actually, I'd be thrilled to hear about this from  
anyone but you seem willing to express opinions that are not just  
apple pie - and that is good if we want to actually change things.

---

From: John C Klensin <[email-masked]>
Date: Fri, 22 Jun 2007 19:00:00 -0400

--On Sunday, 17 June, 2007 22:43 -0700 Lakshminath Dondeti
<[email-masked]> wrote:

>> IMO, you have to have a structure/process/rules that assumes
>> people are generally trying to do the Right Thing. For checks
>> and balances, you then also need appeals procedures and a
>> willingness to speak up and challenge parties when there is
>> evidence of bad decision making.
> 
> Agreed.  I guess I challenge when there is an appearance of
> bad decision making.  I would rather not wait until things get
> out of hand and then attempt use of the hammer of appeals.
> That is just my personal preference.

For reasons associated with, but not identical to, Joel's
reasons for objecting to determinism, I think the position you
express here is ultimately dangerous.

I believe that we should be selecting IESG and IAB members who
can and will exhibit very high levels of technical maturity and
breath, and consistent good judgment.   I believe that being
sufficiently mature and self-aware to avoid either substituting
one's own judgment for that of the community or imposing one's
personal preferences as a blocking requirement is a corollary of
that selection criterion.   If we cannot find enough people like
that, we should shrink the relevant bodies and, if necessary,
the amount of work the IETF tries to do, rather than trying to
get by on third-rate people.

If, being human, IESG or IAB members occasionally slip up, or
need to be asked to consider actions a second time and more
carefully, I think the appeals process is reasonable and
appropriate, not some sort of "hammer" that must be used only in
rare and particularly egregious circumstances.  I note, as
others have noted, that most appeals end at the first stage or
two.  I believe that is a sign of health.  

Similarly, if an IESG or IAB member turns out to have been a
seriously bad choice --because the Nomcom, despite its best
efforts, put someone in place who doesn't come up to the
criteria and expectations above-- or burns out and starts
consistently exhibiting behavior that doesn't meet that
standard, then I think we should, as a community, encourage them
to step down (IMO, one of the reasons we don't see recalls more
often is that, if someone insisted on staying on after having
clearly lost the confidence of the community, whatever Nomcom
made the appointment would have made a _really_ bad decision --
arguably much worse than any we have seen).

So my model is that we give the I* a lot of discretion, we
appoint members of whom we have very high expectations about the
technical ability and general wisdom to apply judgment, and then
we hold them to those expectations.  

If, instead, we reduce that discretion, create and take
advantage for more second-guessing over essentially trivial
things, and try to rely more on procedures (and "determinism")
rather than judgment, we will end up recruiting and appointing
the sorts of people who like working in the environment we
create for them.  And, in the long term, I think that causes a
catastrophe (in the pattern of a number of bodies who operate by
procedure rather than judgment and wisdom.

I like what we have a lot better, even on the days when I feel
obligated to seem to be the voice of obnoxious opposition. 

        john

---

From: Joe Touch <[email-masked]>
Date: Sun, 24 Jun 2007 09:19:12 -0700

Ted Hardie wrote:
...
> That does not mean the IETF leadership is itself a meritocracy; it's not.

I believe there remains a disconnect between what people think the I*
roles are (primarily service, e.g., IMO), and what those in those roles
have sometimes interpreted it as (oversight based on meritocracy).

> "fit" is a critical one. 

Indeed. The primary metric of "fit" means:

	- is willing, available, and *financially* able to serve

Until we remove that last metric - where roles can take upwards of 80%
of someone's time, where letters of support from employers are
requested, if not required, we select from among an increasingly small
and increasingly biased (towards industry participants) subset.

Those selected to serve would serve us all better if they kept that in
mind more often.

Joe

---

From: Randall Gellens <[email-masked]>
Date: Mon, 25 Jun 2007 16:36:12 -0700

>  get by on third-rate people.

I agree with John here, but I see problems with this in practice.  In 
particular, what to one person is clearly substituting one's own 
judgement or preference, is to another just showing good judgement. 
So it is hard to truly select such mature people, because the same 
person will appear to some as mature and to others as inappropriate.

One thing we can try to do is to have more clear criteria for 
exercise of discretion.  Not in the form of hard rules, but simply as 
more detailed guidance that we can hopefully update with experience. 
That makes it easier to determine if someone is abusing discretion.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The nice thing about standards is that there are so many of
them to choose from.                  --Andrew S. Tanenbaum

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 26 Jun 2007 10:19:34 -0500

So, I am curious. Have people looked at 
http://www.ietf.org/u/ietfchair/discuss-criteria.html (which I BELIEVE is 
the current form of the DISCUSS Criteria document - this really needs to be 
an ION, but that's another story)?

Does this look like the kind of "guidance" Randall is talking about?

Thanks,

Spencer

p.s. I tend to agree with many things that have been said in this thread, 
but want to give credit to recent IESGs who have documented things that 
weren't actually written down (DISCUSS criteria, many-but-not-all of BOF 
proposal evaluations, etc.).

From: "Randall Gellens" <[email-masked]>

> particular, what to one person is clearly substituting one's own judgement 
> or preference, is to another just showing good judgement. So it is hard to 
> truly select such mature people, because the same person will appear to 
> some as mature and to others as inappropriate.
>
> One thing we can try to do is to have more clear criteria for exercise of 
> discretion.  Not in the form of hard rules, but simply as more detailed 
> guidance that we can hopefully update with experience. That makes it 
> easier to determine if someone is abusing discretion.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jun 2007 10:26:22 +0200

Joe,

On 2007-06-24 18:19, Joe Touch wrote:
> roles are (primarily service, e.g., IMO), 

That may be your opinion. Mine is that the part which is pure service
is exactly the part we should pay to get done, via contracts and SLAs
with IASA. If we had more money, I could certainly envisage paying for
a full-time Standards Process Manager, to actively manage document
and milestone processing. The part we expect from community members
placed in leadership positions is not that. It's the part that requires
technical breadth and depth, sound judgement, people skills and, er,
leadership.

> and what those in those roles

Maybe, but Nomcom isn't supposed to re-appoint those...

> and increasingly biased (towards industry participants) subset.

Unfortunately I can't see any practical way to change this unless
we decide that instead of 120 WGs we should have, say, 50. I did
tentatively propose splitting the Chair role, but people didn't
bite on that, and it wouldn't solve the load problem for the IESG
as a whole.

    Brian

> ------------------------------------------------------------------------
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jun 2007 06:18:39 -0700

Brian E Carpenter wrote:
> with IASA.

You're saying it's not service, but you're asking for technical
oversight - that's service.

> If we had more money, I could certainly envisage paying for
> placed in leadership positions is not that. 

Agreed; that part is non-technical service, but we seem to spend a lot
of time on process - notably where process procedures don't take the
disconnect between the meritocracy view and the service view in account.

> It's the part that requires

Except for the technical part, that's required of all participants -
paid or not.

> Maybe, but Nomcom isn't supposed to re-appoint those...

I don't see anything that encourages that, let alone requires it.

> bite on that, and it wouldn't solve the load problem for the IESG

We could have more ADs and split and/or layer the work to reduce the
per-person load. That may not be the only - or even best - way forward,
but we need a way forward or we'll just keep coming up with (poor)
excuses for the status-quo.

Joe

---

From: Keith Moore <[email-masked]>
Date: Wed, 27 Jun 2007 09:33:35 -0400

> per-person load. That may not be the only - or even best - way forward,
It's not clearly even a way forward.  the more ADs there are, the harder
it is to coordinate between the ADs and the areas.

IETF structure doesn't scale.   It's not clear that it can scale - at
least, I've never seen a plan for IETF that could scale, and  most of
the ideas I've seen that purport to help IETF scale would make it
worse.   until we find a way to make IETF scale, we have to rely on
prioritizing and load shedding.

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jun 2007 06:52:36 -0700

> it is to coordinate between the ADs and the areas.

Yes, each AD becomes less efficient, but perhaps the goal shouldn't be
the efficiency of the individual volunteers but the ability to be more
inclusive in who we can consider for these roles.

> worse.

The current IETF structure doesn't scale; I proposed a slightly
different one. You can assert that it won't work, but I don't consider
arguing against an assertion constructive.

> until we find a way to make IETF scale, we have to rely on

What we do in the meantime is a different issue.

Joe

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jun 2007 16:49:09 +0200

> inclusive in who we can consider for these roles.

It isn't an issue of individual efficiency. A committee of 15 is already
unwieldy given our desire for consensus; the IESG thought long and
carefully before the last increase (from 6 to 7 Areas). I don't see
how it can be increased further.

One thing that would make a significant difference would be if WGs
really took responsibility for their own quality control. Even at the
trivial level, the IESG still gets drafts that don't pass ID-nits
(but that is getting better, thanks to PROTO shepherding). But maybe
we should be pushing harder to make WGs responsible for getting
final external reviews done (and responded to). That would just be
more delegation along the path that's been followed for the last
several years.

    Brian

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Jun 2007 08:05:33 -0700

> how it can be increased further.

I don't see increasing the areas; I see splitting them down as a
possible way. Leaving an AD at the top level with less work, and having
sub-ADs report to them.

Joe

---

From: Keith Moore <[email-masked]>
Date: Wed, 27 Jun 2007 11:21:28 -0400

> Yes, each AD becomes less efficient
not just that, but at some point the more ADs you add, the less useful
work IESG as a whole gets done.  as of the time I was on IESG, we were
close to that point already.  IMHO.

Keith

---

From: Michael Thomas <[email-masked]>
Date: Wed, 27 Jun 2007 08:42:27 -0700

> more delegation along the path that's been followed for the last

Part of the problem with cross area review is that there's such a flood
of documents that it's pretty impossible to know unless somebody tells/asks
you to review it that you should care. As I said, I'm a little skeptical 
about
"expert review", but for so-called experts as well as the laity in other 
areas,
it might be nice to have some indexing when it touches areas which are
known to contain land mines.

We already do this to some degree with Security Considerations, but they
often read as a "Full Disclosure" part of a contract rather than the 
basis of
how operation security is intended in the draft itself.

What might be nice is to have some cross domain Cliff's Notes capturing
the jist of how this draft uses or affects other areas? So that people 
in other
areas can determine more easily if something whacky might be going on
and pique their interest in reading it? Were this done sooner in the process
rather than later, we well might be able to nip more whackiness before it
bubbles all the way up to the IESG.

       Mike

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 28 Jun 2007 08:33:22 +0200

> possible way. Leaving an AD at the top level with less work, and having
draft-iesg-alvestrand-twolevel, published October 2003, proposed 
something of that sort.

Last trace of discussion I could find with Google was the Geneva IESG 
retreat in 2005 (?), with the quote:

"We discussed whether there was a need to formalize this into a two-level delegation system, 
but concluded that there was no workload advantage in formally changing the primary role of the Area Directors."

I still don't believe that anything significantly less radical will make 
a significant difference to the "IESG workload problem". But at the 
moment, the IESG/IETF interface doesn't seem to be in "immediate crisis" 
mode, so I guess we'll survive another year.

                     Harald

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jun 2007 09:18:28 +0200

On 2007-06-27 20:46, Tony Li wrote:
> It's well known that when dealing with a scalability issue, the way to 
> address the issue is to install hierarchy.  [Have you ever heard me say 
> this before?  ;-)]  Adding another layer of management  is therefore the 
> obvious approach and a fine alternative to having co-ADs or more areas.
> 
> The downside to this is increased inefficiency and disconnect from reality.

I think the other issue is that this layer would be hard to fill in
our volunteer community. Being a WG Chair has its satisfactions, as does
being an AD or IAB member; I'm not quite sure what the satisfactions
in an intermediate position would be. That's why I'd rather push
work and responsibility into the 120 WGs, where there should be
compensating satisfaction in work well done.

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 Jun 2007 10:51:37 +0200

On 2007-06-27 17:42, Michael Thomas wrote:
> you to review it that you should care.

Indeed. The current reviews such as Gen-ART and secdir reviews don't just
happen spontaneously; specific reviewers are triggered by a "dispatcher."

> As I said, I'm a little skeptical 
> about

We also need reviewers to look for land mines in unexpected places.

> bubbles all the way up to the IESG.

It sounds like that would be done before passing the document
on for AD review. Is that your idea?

     Brian

---

From: Joe Touch <[email-masked]>
Date: Thu, 28 Jun 2007 06:10:25 -0700

...
> "We discussed whether there was a need to formalize this into a
> two-level delegation system, but concluded that there was no workload
> advantage in formally changing the primary role of the Area Directors."
> 
> a significant difference to the "IESG workload problem". 

Agreed - perhaps 3-level. ;-)

> But at the

That depends on who you ask; IMO, the dearth of certain communities in
the IESG and IAB has been a continuing cause for concern.

Joe

---

From: Michael Thomas <[email-masked]>
Date: Thu, 28 Jun 2007 06:56:13 -0700

Brian E Carpenter wrote:
>> As I said, I'm a little skeptical about
>> "expert review", but for so-called experts as well as the laity in 
>> other areas,
>> the jist of how this draft uses or affects other areas? So that 
>> people in other
Yes -- well before. Maybe something something along the lines of
what it builds on, might impact, might be applicable to, etc. The
problem that I see is that there a fair amount of beg, borrowing and
stealing (good) and that is often done without the keepers of those things
knowledge until pretty late in the game (bad). Take DNS for example
since it's a favorite franken-building block. Would anybody in the
namedropper crowd be likely to know if, say, the routing crowd
was using DNS in some new way? Probably not. If, however, we
had a dashboard which named out the cross-area dependencies in,
oh say, the area section or the working group page or somewhere
like that it would at _least_ give you a clue that there might be something
interesting to look at in an area you normally pay no attention to.
   
       Mike

---

From: Ralph Droms <[email-masked]>
Date: Thu, 28 Jun 2007 10:08:01 -0400

DHCP is also a frequently-used building block (some would say  
attractive nuisance).  Stig, Jari and I are trying to identify drafts  
from outside the dhc WG that extend DHCP or use DHCP in novel ways,  
so we can provide guidance to the authors of those drafts as early as  
possible.  Jari and Stig have arranged to review the contents of all  
I-Ds to scan for occurrences of "DHCP", and we're following up with  
manual review to identify which of those drafts should be reviewed by  
the dhc WG as a whole.

So far we've identified about 115 drafts that contain "DHCP", and  
have reviewed about 40 of those drafts.  Roughly 10% seem to be  
drafts that the dhc WG should review.

- Ralph

On Jun 28, 2007, at Jun 28, 2007,9:56 AM, Michael Thomas wrote:

>>> "expert review", but for so-called experts as well as the laity  
>>> in other areas,
>>> often read as a "Full Disclosure" part of a contract rather than  
>>> the basis of