Subject: Re: Comments on draft-aboba-sg-experiment-03.txt

From: Bernard Aboba <[email-masked]>
Date: Wed, 10 Oct 2007 12:03:43 -0700 (PDT)

Sam Hartman said:

"It's my understanding of the document that SG formation is subject to
ietf-wide review.  I'm not sure that the IETF review in RFC 2418 nor
the same IETF review for SG formation is quite the same as an IETF LC,
but I do think the same IETF and IAB involvement is anticipated in the
SG process.  The IAB is not involved in proposing SG formation
initially, although they are involved in reviewing whether that is a
good idea before it goes out to the wider IETF community."

That is my understanding as well.  To make it more clear, I would propose 
mentioning IAB review explicitl in Section 3, which is now proposed to 
read as follows:

3.  The Experiment

   This experiment runs for a period of 18 months from IESG approval of
   the experiment.  During the period of the experiment, the IESG MAY
   approve formation of as many as three Study Groups.  The IESG MUST
   inform the community in a public statement of any decisions for Study
   Group formation approved under this experiment.  Such a statement
   SHOULD include a description of specific Study Group that was formed.

   Given that this is an experiment, the intent is for Study Groups to
   be handled identically to Working Groups in terms of IETF process,
   tools and infrastructure; no additional burden is to be imposed on
   the IETF Secretariat.  Other than the abbreviated charter, the
   process for formation of a Study Group is identical to that of a
   Working Group, including review by the IAB and IESG, announcement of
   the potential Study Group, and request for review by the IETF
   community.  The operating rules of a Study Group (openness, meeting
   requirements, etc.) are identical to Working Groups.  From the point
   of view of IETF infrastructure (tools, membership in the WGCHAIRS
   mailing list, process rules, Charter pages, etc.)  Study Groups are
   treated identically to Working Groups, with the exception that Study
   Group names should include "SG" within the name (e.g. "EXAMPLESG"),
   so as to clearly differentiate them from Working Groups.

   Review of Study Group documents will utilize the same tracking tools
   and processes (including PROTO sheparding) as other IETF documents;
   this allows feedback to be viewed by Study Group Chairs and
   participants, as well as providing additional clarity on next steps.
   Formation of a Study Group requires the appointment of a Study Group
   Chair, and a well defined set of Working Group formation criteria
   (agreement on the Charter, review of the formation criteria, problem
   statement or requirements document, etc. )

Eric Rescorla said:

"A related issue is that this puts pressure on ADs to approve
 SGs for efforts that they would ordinarily simply refuse WGs
 for. I.e., "OK, so you won't give us a WG, how about a
 SG". Currently this document simply has it at the IESG's
 discretion"

In order to address the potential for SGs dragging on, I have added a 
default lifetime (6 months), as well as some guidance on SG formation to 
Section 2, which is proposed to read as follows:

"2.  Study Group Formation

   If at any point during the Working Group formation process, including
   after a first or second BoF session, relevance to the Internet
   community and interest within the IETF and end-user community has
   been demonstrated, but one or more Working Group formation criteria
   outlined in RFC 2418 [RFC2418] Section 2.1 has not yet been met, the
   IESG MAY propose that a Study Group be formed.

   Since the goal of a Study Group is to put in place the prerequisites
   for formation of a Working Group more rapidly than might otherwise be
   possible, Study Groups SHOULD initially be chartered for a period of
   six months to twelve months, with six months being the default.
   While the IESG MAY extend the initial Study Group milestones by an
   additional six months, extensions beyond this are NOT RECOMMENDED.
   The Charter for a Study Group SHOULD include at least the following
   "basic milestones":

      o Development of a Working Group Charter.

      o Development of a document demonstrating fulfillment of
        the Working Group formation criteria described in
        RFC 2418 [RFC2418] Section 2.1.

   The IESG MAY also include additional milestones within a Study Group
   charter (such as development of a problem statement or requirements
   document and/or completion of a review of the literature or current
   practices), as long as these additional milestones do not compromise
   the ability of the Study Group to deliver on the basic milestones in
   a timely way.  A Study Group charter MUST NOT include milestones
   relating to development of standards track documents or protocol
   specifications.

   Since the Study Group experiment is not intended as a substitute for
   the existing Working Group formation process, Study Groups SHOULD be
   formed only in situations where the prerequisites for formation of a
   WG are likely to be met if the SG successfully completes the basic
   milestones."

With respect to success metrics, I would propose to add the following 
Section 3.1:

"3.1.  Success Metrics

   Since one of the goals of this experiment is to enable the more rapid
   formation of Working Groups, the success of an individual Study
   Group, as well as the experiment, can be measured based on the
   progress made toward Working Group formation.  Useful metrics
   include:

Progress on Basic Milestones
     A Study Group that does not make progress on its basic milestones
     cannot be judged successful, regardless of its other achievements,
     such as progress on a literature review or requirements document.
     Progress on the basic milestones is measured by whether they are
     completed within the time-frame specified in the initial Study
     Group Charter, and whether feedback from the IESG, IAB and IETF
     community is positive, leading the IESG to vote to form a Working
     Group.

Mailing List Activity
     Since one of the goals of the Study Group experiment is to avoid a
     potential loss of interest among participants, evidence of
     continued engagement on the part of Study Group participants based
     on mailing list activity is a potential success metric.
     Conversely, a Study Group whose mailing list shows minimal traffic
     would probably not be a good candidate for milestone extension."

The complete text of the strawman -03 document is available here:
http://www.drizzle.com/~aboba/IAB/draft-aboba-sg-experiemnt-03.txt

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Spencer Dawkins <[email-masked]>
Date: Wed, 10 Oct 2007 22:08:36 -0500

Hi,

> http://www.drizzle.com/~aboba/IAB/draft-aboba-sg-experiemnt-03.txt

Easily corrected typo, but just for ease of clicking, the correct URL is

http://www.drizzle.com/~aboba/IAB/draft-aboba-sg-experiment-03.txt

Now, for something completely different...

Since this is an experiment - please don't hold up the experiment while you 
try to legalese-craft all the corner cases, but you might include a "things 
to clarify if the experiment succeeds" list... which might include this 
question...

>From way down at the bottom of the 03 Introduction (which is really long, 
but):

   This document describes an RFC 3933 [RFC3933] experiment in the
   Working Group formation process, known as the Study Group.  Study
   Groups MAY be formed by the IESG when there is evidence of clear
   interest in a topic on the part of IETF participants and end-users,
   and relevance to the Internet community has been demonstrated, but
   other RFC 2418 [RFC2418] criteria relating to Working Group formation
   (including creation of a satisfactory Charter) have not yet been met
   as the result of a first or second Birds-of-a-Feather (BOF) session.

So far, so good... I'm not confused until the next paragraph:

   Study Group milestones are focused on completion of prerequisites for
   Working Group formation, and as a result they are expected to
   conclude within a short time frame, with limited opportunities for
   milestone extension.  This Study Group experiment does not alter the
   Working Group formation guidelines described in RFC 2418 [RFC2418]
   Section 2.1, the processes relating to BoFs [BOF]  or the Internet
   Standards Process described in RFC 2026 [RFC2026].

Is everyone but me totally clear on how BOFs interact with SGs and WGs?

The way I'm reading this, the mainline path through this procedure is that 
some community of interest requests a BOF, with a request that's plausible 
enough for an AD to go for it, and then the IESG suggests a SG after the 
BOF.

If the SG "succeeds", is there any opportunity to hold a second BOF (which 
seems reasonable, as a WG-forming BOF that would have more IETF-wide 
visibility), or does the SG have to go straight to WG (which is the way I 
read version 03)?

And, for extra credit, if the answer is "yes", what if the community held 
two BOFs before the SG formed (which would be the 2418 limit)? The answer 
based on "does not alter the process" seems to be "no" - is that what we 
want?

And a couple of nits...

Nit: there are places in this draft that say "charter", but since both SGs 
and WGs have charters, it would be great if these occurences were qualified 
as "SG charter" or "WG charter".

Nit: [BOF] is great advice (I've reviewed it at least once for Thomas), but 
it's not "the processes relating to BOFs [BOF]". The last time I saw [BOF], 
it was intended to be informational - the process is still normatively 
described in 2418.

Thanks,

Spencer

---

From: Lakshminath Dondeti <[email-masked]>
Date: Wed, 10 Oct 2007 22:08:51 -0700

Given the confusion around this, let me try to enumerate all paths

(I will enumerate the end result as WG, but please substitute it with 
"Stop" for the failure cases)

Idea -->  WG
Idea -->  SG --> WG
Idea -->  BoF-1 --> WG
Idea -->  BoF-1 --> BoF-2 --> WG
Idea --> BoF-1 --> SG --> WG
Idea --> BoF-1 --> BoF-2 --> SG --> WG

I am not too keen on SG --> BoF; but, if the sponsoring AD wants to do 
it, I guess it may make sense to not explicitly disallow it.   We can 
probably look for that data during the experiment.  If the SG failed to 
develop a consensus charter during its life, a BoF session most likely 
won't result in a different result (although one could argue that due to 
time constraints or other reasons, the community evaluation of the 
charter they developed can be done at a f2f meeting as opposed to at a 
meeting; a similar argument has been made here by folks).  To not flout 
the 2418 rule, if two BoFs were already held and an SG was created, it's 
WG or nothing after the SG.

So, we may also accommodate

Idea --> SG --> BoF-1 --> WG
Idea --> BoF-1 --> SG --> BoF-2 --> WG

The following two options do not make sense:
Idea --> SG --> BoF-1 --> BoF-2
Idea --> BoF-1 --> SG --> BoF-2 --> BoF (2418 disallows this)
Idea --> BoF-1 --> BoF-2 --> SG --> BoF (2418 disallows this)

(Hope I didn't make a mistake there ;) )

regards,
Lakshminath

On 10/10/2007 8:08 PM, Spencer Dawkins wrote:
> Since this is an experiment - please don't hold up the experiment while 
> you try to legalese-craft all the corner cases, but you might include a 
> "things to clarify if the experiment succeeds" list... which might 
> include this question...
> 
> The way I'm reading this, the mainline path through this procedure is 
> that some community of interest requests a BOF, with a request that's 
> plausible enough for an AD to go for it, and then the IESG suggests a SG 
> after the BOF.
> 
> If the SG "succeeds", is there any opportunity to hold a second BOF 
> (which seems reasonable, as a WG-forming BOF that would have more 
> IETF-wide visibility), or does the SG have to go straight to WG (which 
> is the way I read version 03)?
> 
> And, for extra credit, if the answer is "yes", what if the community 
> held two BOFs before the SG formed (which would be the 2418 limit)? The 
> answer based on "does not alter the process" seems to be "no" - is that 
> what we want?
> 
> And a couple of nits...
> 
> Nit: there are places in this draft that say "charter", but since both 
> SGs and WGs have charters, it would be great if these occurences were 
> qualified as "SG charter" or "WG charter".
> 
> Nit: [BOF] is great advice (I've reviewed it at least once for Thomas), 
> but it's not "the processes relating to BOFs [BOF]". The last time I saw 
> [BOF], it was intended to be informational - the process is still 
> normatively described in 2418.
> 
> Thanks,
> 
> Spencer
> 
>

---

From: Eliot Lear <[email-masked]>
Date: Thu, 11 Oct 2007 13:38:02 +0200

I have one additional concern about this proposal.  If a study group is
intended to meet at an IETF, it will compete with slot requests both
from IETF working groups and IRTF research groups.  I wouldn't want to
prohibit f2fs but I would certainly suggest that they come in low on the
totem pole for space requests.

Eliot

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 11 Oct 2007 15:54:31 -0700

Would you see them being above or below BOFs?

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 11 Oct 2007 16:10:16 -0700 (PDT)

The number of slots taken up by SGs is controlled by how many SGs are 
chartered  and how long the SGs take.  Both factors are under the control 
of the IESG, which has already suggested a maximum of 3 GS to be chartered 
under the experiment. 

Assuming that the IESG charters SGs for the minimum period initially (6 
months), is tough on extensions, and balances the value of SGs with 
the available room slots and other factors, the problem seems 
controllable, particularly since a fair number of WGs are on the verge of 
concluding.

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 11 Oct 2007 16:35:42 -0700

Sure, I agree with all that but I was wondering how people would view  
the SG vs BOF issues. I know anything will be a judgement call with  
tradeoffs - I'm sure we have had occasion where running the BOF was  
more important than some of the WGs but as for the most part WG get  
strong precedence over BOFs. I'm wondering if the SGs are going to  
make it even harder to schedule BOF of if the SGs should expect to  
get 100% bumped when we have a meeting that wants a few BOFs in one  
of the fuller areas. Clearly neither of these sounds much fun but  
given I sometimes have to help make theses scheduling tradeoffs, I  
would love to have any input from the community on what I should be  
doing.

---

From: Eliot Lear <[email-masked]>
Date: Fri, 12 Oct 2007 09:14:40 +0200

Cullen Jennings wrote:
> Would you see [SG meetings] being above or below BOFs?

Well now that's the question, isn't it?  I'd be curious as to what
Bernard thinks.

Eliot

---

From: Jari Arkko <[email-masked]>
Date: Sat, 13 Oct 2007 12:13:36 -0500

Eliot,

>> totem pole for space requests.

Yes.

I think I'd prefer to make the decision about having capacity for new
officially recognized efforts (BOFs, SGs, WGs) at time that we're accepting
these efforts. For instance, on my half of the area, I'd like have no more
groups than what I can personally attend, without being in two or more
places at the same time.

(But lets not de-prioritize SGs (or BOFs) out of the meeting completely.
The point of an official effort is that it gets attention, resources, etc;
A judgment call by the AD/IESG/IAB/community that this attention
is deserved in this particular case.)

Jari

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 14 Oct 2007 15:38:32 +0300

Hi Bernhard,
Hi Lakshminath,

I read through draft-aboba-sg-experiment-03.txt and have a few comments.
I had a quick reply written and then I deleted everything again because 
I mainly struggled with two questions:

* Pros/Cons of the SG

+ Some (or all) existing tools could be reused
(Could also be done without a SG as well.)
+ Formal status within the IETF
(Could be valuable in some interactions with other SDOs)
- Risk of delaying work further and introducing even more formalism to 
the IETF.

* What are the problems a SG could solve?

The document is quite short on these aspects; it only says:
"
   In some situations, while interest on the part of IETF participants
   and end-users may be evident, and the relevance to the Internet
   community may be demonstrated, the answer to other questions (such as
   an understanding of existing work, clarity or achievability of goals,
   or overlap with existing working groups or standards bodies) may not
   be as clear.
"

I really wonder whether these are truely the problems for unsuccessful 
BOFs.
It would have been good to have meta-discussions after a BOF about why 
it wasn't successful and suggestions on how todo it better next time.
The reasons for lack of success in a BOF are manifold. Some of them are 
listed in 
http://www.ietf.org/internet-drafts/draft-narten-successful-bof-03.txt.
I am not convinced that a SG will help to improve something per-se.

Last Thursday I attended the SAVA meeting in Helsinki (see 
http://mail.nrc.tsinghua.edu.cn/pipermail/sava/2007-October/000211.html). 
It was organized by Jari Arkko as an effort to help unexperienced guys 
with their attempt to bring some work to the IETF. (Note that's my own 
interpretation of Jari's intention here. He might have had other ideas 
in mind.) Sometimes it just seems to be better for some experienced IETF 
persons (e.g., IAB, IESG or other individuals) to help people with a BOF 
preparation effort. To me this seems to be far more efficient 
particularly since many BOFs fail due to the lack of "street 
credibility" by the folks initiating the work. Furthermore, I am not 
sure whether to start a SG after a BOF is the right thing todo. I 
personally would think that helping people earlier would be useful 
(e.g., after one or more Bar-BOFs).

Hence, I wonder whether it would be useful todo the following:

* Give the concept a name. "Study Group" is fine with me.
* Start with with SG as early as possible; no need to wait for a failed BOF
* Reuse the IETF tools (mailing lists, web pages, etc.).
* Find a few knowledgeable IETF persons (who are interested to help -- 
and not delay or destroy efforts) and assign them as mentors to the 
preparation work.
[Note: If you cannot find any experienced person to help then that's 
probably already an indication that something is really heading into the 
wrong direction.]
Rather than putting this into the constraints of a working group I would 
intentionally leave this a bit "unregulated". This gives the 
participants more freedom to organize conference calls, f2f meetings, 
etc. and to announce them on the list without having long debates about 
whether it was sent with enough time in advance. Additionally, it would 
leave the group open to come up with other innovative ways to make progress.

For example, at IETF#69 we had an adhoc meeting on SPIT prevention. In 
short, it was a disaster.
We have a few solution approaches, we can envision a problem but we 
obviously do not have a lot of SPIT to determine whether the proposed 
mechanisms would help and which mechanism is more likely to be 
successful. The group of people working on the subject would obviously 
appreciate input from other IETF members on how we should move forward 
with the work since the work stalled a bit. This is a quite practical 
example and I would like to better understand how the proposals in 
draft-aboba-sg-experiment-03.txt could help me to make progress.

I could also come up with a simpler example, based on the early warning 
adhoc meeting from IETF#69. It is a simpler case since we had 
constructive discussions and there was a lot of support for the work. 
Let me know if I should elaborate a bit more on this case if the 
above-mentioned one is too tough.

In any case, if you come up with ways to work more efficiently then 
that's certainly great.

Ciao
Hannes

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 14 Oct 2007 11:48:30 -0700 (PDT)

> I am not sure whether to start a SG after a BOF is the right thing todo. 

Depending on the outcome of the BOF, it may or not be.  A SG can only help 
if a problem important to the Internet community has been identified, if 
the problem is well understood, and if there is sufficient interest/energy 
to take on the work. 

> I personally would think that helping people earlier would be useful 

The SG experiment permits that at the AD's discretion.  Personally, I 
would like to see how well this type of SG would work. 

> [Note: If you cannot find any experienced person to help then that's probably
> already an indication that something is really heading into the wrong
> direction.]

This is part of the required "demonstration of interest" for SG formation, 
I would think. 

> For example, at IETF#69 we had an adhoc meeting on SPIT prevention. In short,
> it was a disaster.
>
> We have a few solution approaches, we can envision a problem but we obviously
> do not have a lot of SPIT to determine whether the proposed mechanisms would
> help and which mechanism is more likely to be successful. 
> This is a quite practical example and I would like to better understand
> how the proposals in draft-aboba-sg-experiment-03.txt could help me to make
> progress.

>From your description, it seems that there is not yet enough data to 
develop a problem statement, let alone to evaluate solutions.  In such a 
situation, it seems that the most that could be accomplished would be to 
do a literature review, summarizing the state of the art and perhaps 
identifying problems for further research.  I'm not clear that a SG would 
be the best avenue for that, due to the short duration.  An IRTF RG might 
make more sense. 

> I could also come up with a simpler example, based on the early warning adhoc
> meeting from IETF#69. It is a simpler case since we had constructive
> discussions and there was a lot of support for the work. 

Of of the top of my head, this sounds like it could potentially be a 
better understood problem, for which the SG might be more relevant.

---

From: Hannes Tschofenig <[email-masked]>
Date: Mon, 15 Oct 2007 11:19:24 +0300

Hi Bernhard,

~snip~
> develop a problem statement, let alone to evaluate solutions.

Right. As you know from the Email spam the problem is only that waiting 
longer makes the problem substantially harder to solve later.
In some sense this is similar to security in general.

To make it more complicated it might not provide you a lot of additional 
insight when you know how the current attacks look like. The future ones 
will be different  anyway.

>   In such a 
> do a literature review,
We did this already. Most existing documents just say what has been 
stated in various IETF drafts already.

>  summarizing the state of the art and perhaps 
> identifying problems for further research.

I am not sure whether it is such a big research topic. If you, for 
example, look at XMPP then these folks have already published a lot of 
their specifications to deal with this subject. They are far ahead of 
the IETF crowd. They just do it and then figure out whether it helps or 
not. The worst thing that can happen is that it does not get implemented 
when people don't think it isn't useful.

>   I'm not clear that a SG would 
I am not so sure about the value of IRTF groups.

> better understood problem, for whichthe SG might be more relevant. 
>   
Good to know. Maybe we have a potential candidate for a SG experiment here.

I guess I would also like to see the outcome of this experiment. I 
wonder what you think about my other suggestions that go more along the 
lines of "providing help from experienced people"? Do you think that 
this will automatically happen as soon as a SG is formed? I don't think so.

Ciao
Hannes