Subject: [admin-discuss] Fwd: Draft report on first annual IETF community survey

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 09 Jul 2021 13:31:37 +1000

First comment: there are no numbers on the sections, so it's hard to specify locations for feedback :p

Overall comment.  Ouch - 56,000 participants of which only 2,032 valid responses.  4% response rate is tiny, and makes the "maximum margin of error of +/- 2.13%" very questionable.  That would be true IFF this 4% is a random sample, but that is not a given.

On the other hand, 56,000 list subscribers is going to have a large number of lurkers - if somebody hasn't posted to any mailing list in the past 5 years, submitted any draft, or attended any meeting or interim call, it would be reasonable to assume that they are not at all active in the IETF any more.  Merely being subscribed to a mailing list and not interacting is no different than reading the archive online or via anonymous IMAP!

I would be interested in having a breakdown of how many "active" participants by some reasonable standard the IETF has.  I wonder if one takeaway from this report should be that the IETF should require some affirmative step to become a "participant" that needs to be renewed every few years - datatracker account and notewell acknowledgement, so that people drop out through lack of interaction unless they take a step to affirm that they are participating in the IETF. 

Section: "The IETF has a big problem with gender, a smaller problem with regionality but
no problem with age" the final paragraph appears to be incomplete.  It finishes with ". Some people" and then just cuts off.

----- Original message -----
From: IETF Executive Director <[email-masked]>
To: IETF Announcement List <[email-masked]>
Subject: Draft report on first annual IETF community survey
Date: Friday, July 09, 2021 06:46

In May of this year the IETF Administration LLC (IETF LLC) on behalf of the IESG and in collaboration with the IAB distributed the first annual IETF community survey to all 56,000 addresses subscribed to IETF mailing lists.  As noted in the blog post [1] that launched the survey, its purpose was "To help better understand our community and its makeup, gather views on the IETF and how well it works for participants, and gain insight into how we compare to similar organisations".  We want to thank the 2032 people who responded to this survey, providing a rich data set.

A report has now been written jointly by the IETF LLC and IESG that analyses the results and draws some key findings from the analysis. As this is the first year of this survey, we are seeking feedback on a draft version of this report before a final version is published.

The draft report is available at:

    https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf

Please provide any feedback on the draft report to the [email-masked] mailing list or directly to the IESG at [email-masked] by 9 August 2021.

Please also feel free to reach out to me directly at [email-masked] if you have any questions or concerns.

[1] https://www.ietf.org/blog/first-annual-ietf-community-survey/ 

-- 
Jay Daley
IETF Executive Director
[email-masked]

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf-announce

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Jay Daley <[email-masked]>
Date: Fri, 9 Jul 2021 16:08:49 +1200

Thanks Bron

> On 9/07/2021, at 3:31 PM, Bron Gondwana <[email-masked]> wrote:
> 

Noted for the final version.

> 
> Overall comment.  Ouch - 56,000 participants of which only 2,032 valid responses.  4% response rate is tiny, and makes the "maximum margin of error of +/- 2.13%" very questionable. 

That’s a matter of maths not opinion, which is at https://www.surveysystem.com/sscalc.htm

> That would be true IFF this 4% is a random sample, but that is not a given.

That’s not quite how it works.  The sampling bias in this case is towards people who interact with the IETF as they have clearly answered at a higher rate then those that do not.  But that doesn’t mean the margin of error is wrong unless we can see that they answer their questions differently from those that don’t and in a few places that has been checked and there is no difference.

> I would be interested in having a breakdown of how many "active" participants by some reasonable standard the IETF has.  I wonder if one takeaway from this report should be that the IETF should req…

The survey attempts to do that by defining an interaction status based on self-reported actions.  That is then used for analysing some of the results.  Next time around we need to do more on that, probably with a "in the last year", "in the last five years" type breakdown of the interaction question.
> no problem with age" the final paragraph appears to be incomplete.  It finishes with ". Some people" and then just cuts off.

Noted.

thanks again
Jay

> From: IETF Executive Director <[email-masked] <mailto:[email-masked]>>
> To: IETF Announcement List <[email-masked] <mailto:[email-masked]>>
>     https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf <https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf>
> 
> Please provide any feedback on the draft report to the [email-masked] <mailto:[email-masked]> mailing list or directly to the IESG at [email-masked] <mailto:[email-masked]> by 9 August 2021.
> 
> Please also feel free to reach out to me directly at [email-masked] <mailto:[email-masked]> if you have any questions or concerns.
> 
> 
> [1] https://www.ietf.org/blog/first-annual-ietf-community-survey/ <https://www.ietf.org/blog/first-annual-ietf-community-survey/> 
> 
> -- 
> Jay Daley
> IETF Executive Director
> [email-masked] <mailto:[email-masked]>
> 
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/ietf-announce <https://www.ietf.org/mailman/listinfo/ietf-announce>
> 
> 
> --
>   [email-masked] <mailto:[email-masked]>
> 
> 
> -- 
> admin-discuss mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/admin-discuss <https://www.ietf.org/mailman/listinfo/admin-discuss>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 9 Jul 2021 20:52:24 +0100

I think a couple of the (presumably tentative) conclusions
are making unwarranted assumptions. Just because younger
IETF participants may prefer IM or issue trackers today does
not mean those specific technologies will be preferred by
anyone in future - people can change their minds and their
workflows and better new tooling might come along.

Cheers,
S.

---

From: Jay Daley <[email-masked]>
Date: Sat, 10 Jul 2021 08:46:20 +1200

> On 10/07/2021, at 7:52 AM, Stephen Farrell <[email-masked]> wrote:
> 
> 

Yes I agree that commentary was a step too far. I’ll remove for the final report. 

Jay

-- 
Jay Daley
IETF Executive Director

> Cheers,
> S.
> <OpenPGP_0x5AB2FAF17B172BEA.asc>
> -- 
> [email-masked]
> https://www.ietf.org/mailman/listinfo/admin-discuss

---

From: Jay Daley <[email-masked]>
Date: Sat, 10 Jul 2021 09:06:27 +1200

Hi Bron sorry my answer was a bit short and oversimplified. 

Yes it’s true that the margin of error only holds if this is a random sample. We don’t have an idea of the sizes of most of the sub-populations so we can’t test against those. The sub-population sizes we do have are all biased towards those who interact more.   For example, we have good data on residential region for meeting participants, but nothing on age or gender. And we have the questions about what interactions people have had and what of the three general mailing lists they subscribe to. Cross checking by what data we have seems to show that the sample is a fair sample of those that interact, and even then that’s only at the level of each question so there’s an assumption that there’s no bias in that group for a specific question and not others. But we can’t test if it is representative of those who do not interact much. 

If you think we should give a better warning about the limits of interpretation then I would welcome suggested text as I’m probably too close to this to get that right.

Jay

-- 
Jay Daley
IETF Executive Director

> On 9/07/2021, at 4:09 PM, Jay Daley <[email-masked]> wrote:
> 
> Thanks Bron
>

---

From: Jay Daley <[email-masked]>
Date: Mon, 26 Jul 2021 10:15:08 +1200

> On 10/07/2021, at 8:46 AM, Jay Daley <[email-masked]> wrote:
> 
> 
> 
> 

I plan on amending the text from

From Q24 we learn that support for new modes of participation is limited in comparison to email and that there is a strong inverse correlation between age and strength of support.  If we assume that strength of support will increase as older people cease participating and younger people join, it looks like it will be about 5-10 year for issue trackers to become mainstream and 20-30 years for instant messaging.

to 

From Q24 we learn that support for new modes of participation is limited in comparison to email and that there is a strong inverse correlation between age and strength of support.  Future surveys will tell us if people maintain these preferences as they age and if the IETF should therefore be planning for this change.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]