Subject: Re: [admin-discuss] Draft report on first annual IETF community survey

From: Cat Okita <[email-masked]>
Date: Thu, 8 Jul 2021 14:13:24 -0700

Hola!

re:
> 	30) It is perhaps surprising that almost 90% monitor one or more mailing lists (66% 2 or more) without participating. It would be useful to explore this in future to understand more about why that is and if this can be made easier.

That's about the standard %age you'd expect for participation -- 1 out of 10.  I don't have links immediately to hand, but the topic has been researched for politics as well as commerce (probably best known as "For every 1 person that complains, there are X others with the same complaint"), and there should be more detailed analyses available.

cheers!

> On Jul 8, 2021, at 13:46, IETF Executive Director <[email-masked]> wrote:
> In May of this year the IETF Administration LLC (IETF LLC) on behalf of the IESG and in collaboration with the IAB distributed the first annual IETF community survey to all 56,000 addresses subscribed to IETF mailing lists.  As noted in the blog post [1] that launched the survey, its purpose was "To help better understand our community and its makeup, gather views on the IETF and how well it works for participants, and gain insight into how we compare to similar organisations".  We want to thank the 2032 people who responded to this survey, providing a rich data set.
> 
> A report has now been written jointly by the IETF LLC and IESG that analyses the results and draws some key findings from the analysis. As this is the first year of this survey, we are seeking feedback on a draft version of this report before a final version is published.
> 
> The draft report is available at:
> 
>    https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf
> 
> Please provide any feedback on the draft report to the [email-masked] mailing list or directly to the IESG at [email-masked] by 9 August 2021.
> 
> Please also feel free to reach out to me directly at [email-masked] if you have any questions or concerns.
> 
> 
> [1] https://www.ietf.org/blog/first-annual-ietf-community-survey/ 
> 
> -- 
> Jay Daley
> IETF Executive Director
> [email-masked]
> 
> 
> 
> _______________________________________________
> IETF-Announce mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf-announce

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 9 Jul 2021 11:41:27 +1200

Hi,

This is very interesting, thanks. One immediate comment:

> The standout exception is the timely production of RFCs, judged as poor in both Q25
> and Q27.

I think this summary is a bit misleading. The complaint is surely not about the "production of RFCs" but about the total time from -00 individual draft to the final publication of the RFC. As we know, the actual production of the RFC is a very small fraction of the total time. At best, the final item in Q25 was unclear on this, and Q27 was explcitly about WG process.

So I think the summary should be more likely:

"The standout exception is the slowness of the entire IETF process, judged as poor in both Q25 and Q27."

Also, 2039 responses for >50k targets is a bit lower than I'd have liked. Did you have any way to analyse how representative these 2039 people might be?

Regards
   Brian

---

From: Jay Daley <[email-masked]>
Date: Fri, 9 Jul 2021 11:54:59 +1200

> On 9/07/2021, at 11:41 AM, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi,
> 

Yes, that’s better, thanks.

> Also, 2039 responses for >50k targets is a bit lower than I'd have liked. Did you have any way to analyse how representative these 2039 people might be?

Yes it is lower than I hoped for too.  I think I probably should have narrowed the delivery list to the subscribers to mailing lists that have had at least one message in the last year but I don’t know how much that would reduce the 56k.

The only measures we have for representativeness are those in the survey.  Did you have anything else in mind?

Jay

> admin-discuss mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/admin-discuss <https://www.ietf.org/mailman/listinfo/admin-discuss>
-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 9 Jul 2021 14:26:12 +1200

On 09-Jul-21 11:54, Jay Daley wrote:
> 
> 
>> On 9/07/2021, at 11:41 AM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> Hi,
>>

I'm no expert on surveys & sampling bias. I just wondered if there is any 
objective difference between what we know about the 2k who responded and what we know about the whole 50k. Even very crude data like "% of responders with XXX.com addresses" vs "% of all targets with XXX.com addresses" for a few interesting values of XXX. If a few metrics like that are similar for the 2k and the 50k, it would raise confidence that the results are 
representative.

   Brian

>>>    https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf <https://www.ietf.org/media/documents/REPORT_All-of-IETF_Survey_2021_Consultation_Draft.pdf>
>>>
>>> Please provide any feedback on the draft report to the [email-masked] <mailto:[email-masked]> mailing list or directly to the IESG at [email-masked] <mailto:[email-masked]> by 9 August 2021.
>>>
>>> Please also feel free to reach out to me directly at [email-masked] <mailto:[email-masked]> if you have any questions or concerns.
>>>
>>>
>>> [1] https://www.ietf.org/blog/first-annual-ietf-community-survey/ <https://www.ietf.org/blog/first-annual-ietf-community-survey/> 
>>>
>>
>> -- 
> [email-masked] <mailto:[email-masked]>
>

---

From: Jay Daley <[email-masked]>
Date: Fri, 9 Jul 2021 15:59:40 +1200

> On 9/07/2021, at 2:26 PM, Brian E Carpenter <[email-masked]> wrote:
> 

The survey was anonymous so the only data we have is that supplied.  

Sampling bias is determined by matching to known datasets, of which we have a few.

Clearly, there is some sampling bias as shown by the percentages of respondents that are subscribed to various lists.  By other measures though, such as region, it appears not.

None of this should be confused with being representative as a sampling bias does not equate to a lack of representation unless we know that sample has different views.  Otherwise it’s simply a matter of maths and margin of error.  In my view this data is sufficient for the conclusions to be valid.

Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]

---

From: tom petch <[email-masked]>
Date: Mon, 12 Jul 2021 11:12:35 +0100

> [1] https://www.ietf.org/blog/first-annual-ietf-community-survey/

A left field thought.  I am struck by the fact that on demographic 
questions, I am in a small minority and it would only take a few such to 
identify me uniquely.  I would have seen my concerns about privacy as 
being low compared to the views expressed and the work being done by 
others in the IETF but lately, I have received many phishing e-mail from 
addresses in the domain of my ESP which seem to be derived from posts 
that I have made to IETF lists and that turns my concerns up a notch. 
So, in future such surveys, from the IETF and others, I expect that I 
will make more use of 'Not prepared to say' or just take a guess as to 
the most likely answer by others and say that applies to me.  I wonder 
if privacy concerns colored the answers, or lack thereof, of others.

I don't think that there is anything that the IETF can do to reduce 
these concerns; the way the IETF works leaves it open to bad actors 
harvesting active e-mail addresses and phishing.

On the other hand, rfc-interest is debating the value of postal 
addresses; I cannot seem myself providing one of those again.

Tom Petch

---

From: tom petch <[email-masked]>
Date: Mon, 12 Jul 2021 12:27:04 +0100

p.4 'big problem with gender'

seems to be curtailed at 'Some people'

Tom Petch