IETF Community Survey 2023

In December 2023 the third annual IETF community survey was distributed to all ~49,000 addresses subscribed to IETF mailing lists.   This survey aims to deliver three outcomes:

• A current size and demographic breakdown of the IETF community.
• Data to inform the IETF community, particularly those in leadership roles, on what are some of the key issues affecting the IETF and why sometimes asserted issues are not actually issues.
• A step in a time series of data that can be used to assess the natural changes affecting the IETF and the effectiveness of major programs, organisational changes and community/leadership actions.

A draft report was published for consultation in early March 2024 and minor changes made following feedback received, leading to today's publication of the final report.

A special thanks to the 2000+ people who responded to this survey, providing a rich data set.

The key findings section of the report is reproduced below:

Key Findings

It is important to begin this section with the observation that there are multiple questions where the overall result is strong and indicates there is no significant problem that needs to be looked at, but cross-tabulation by one or more demographic factors shows that one or more groups within that have a much less positive view.  This is sometimes referred to as the tyranny of the majority.

The IETF is good at delivering its mission and principles but could do much better

The answers to Q25 show that IETF participants believe the IETF produces high quality, relevant RFCs in an open, transparent and consensus process. 

The key areas for improvement in Q25 are shown by the lower ratings for “The IETF focuses on the most important work” and “IETF processes are effective”, and a much lower rating for “The IETF produces RFCs in a timely manner”.  These match the low ratings for “WG decisions take a reasonable amount of time” and “WGs meet their goals” in Q27.  

The IETF is considered a very important organisation that outperforms its peers in all respects

Q28 asks how important the IETF is for the development of the Internet using Net Promoter Score (NPS) methodology and the NPS score of +56% indicates that participants consider it to be very important.

Q30 asks respondents to compare the IETF to other SDOs for multiple aspects and overall.   On all aspects: openness, fairness, barriers, quality, cost, administration, behaviour and speed, the IETF is rated better than other SDOs.  The only negative is in the breakdown of Q30 by participant type, which shows that regular IETF participants rate the behaviour of IETF participants as worse than for other SDOs.

Participation is driven by personal interest and a specific technology, not business or employer goals

In Q12 participants rate the most important reasons for participation as those related to personal interest and development, with those related to business and employment much lower.  In Q11a, which is only asked of new participants, the most important reasons for people starting to participate are all related to specific technologies (though this question has a high rate of “None of the above” answers) and those related to employers are rated lower.

In Q27a, “I learn a lot from participating in WGs” is rated the truest of the provided statements about individual participation in WGs.

The IETF still has a problem with gender diversity but there are signs of improvement

The percentage of respondents who identify as women in Q4, 7.84% (between 5.58% - 10.10% by the margin of error), is very low compared to the percentage in the general population.  No benchmarking is presented as multiple participants have commented that the best benchmark is with the percentage of women in engineering roles rather than general IT roles, and that data has not been sourced.

In multiple questions, women report a worse experience of the IETF than men.  In Q26 women report a significantly lower rating for “I am treated the same as the rest of the IETF community”.  In Q26a women report a significantly higher degree of hindrance from “The behaviour of IETF participants” and “The diversity profile of the IETF community”.  The one positive to note is that the percentage of women in leadership roles, as shown in Q7b, is significantly higher than their percentage overall.

The trend from Q11 indicates that the percentage of women in recent annual cohorts is significantly higher than the current overall percentage.  Q4 does not show a disproportionate number of women in previous participants and so it is reasonable to assume that this change in the intake of participants will lead to a change over time.

Participation is dominated by people from just two regions and there are multiple factors behind that

Q2 shows that North America and Europe together account for approximately 40% each of all IETF participants, with Asia a distant third at 11%.  

Participants from other regions report various factors that adversely impact their participation.  In Q5, participants from Asia and Latin America report much lower levels of skills with English, particularly oral skills, and in Q26 participants from those two regions give much lower ratings, only in the Acceptable range, for “My skill in English is good enough for me to participate fully”.  Also in Q26, participants from Latin America give much lower ratings for “I understand the IETF well enough to participate fully” and “I feel part of the IETF community”. Q26a presents a number of additional factors, particularly the cost of participating in IETF meetings (Asia, Africa, Latin America), the complexity of IETF processes (Africa, Latin America) and the reliance on email and mailing lists (Asia, Africa and Middle East).

There is a problem with behaviour but it is hard to pin down

The breakdown of Q30 by participant type shows that regular IETF participants rate the behaviour of IETF participants as worse than that in other SDOs. This correlates with the answers this same cohort gives in Q26a about “The behaviour of other IETF participants”.  Also, in Q26a it appears that the more someone is engaged with the IETF (by participant type) the lower they rate the behaviour of participants.  Finally in Q26a, there is a significant difference in views between men and women about how much of a hindrance, behaviour is.  Together these indicate that there is a problem with participant behaviour.

Q27 and Q27a ask about perceptions of WGs and personal participation in WGs and while behaviour is rated lowest for regular participants, overall it is middling.

New participants need to learn a lot to be effective and without that, people can feel excluded

In Q26 new participants give very low ratings for “I understand the IETF well enough to be effective” and “I feel part of the IETF community”, much lower than other participant types. Those who have not participated in an IETF meeting (onsite or remote) rate these two much lower than those who have.

In Q26a new participants identify “The complexity of IETF processes” as their biggest hindrance to participation, more than any other hindrance identified by any participant type. Then in Q27a new participants give the lowest ratings to “I am able to share my views in WGs” and “My contributions to WGs are valued”.

Q18 asks previous participants to select their reasons for ceasing participation and “I was unable to get my ideas adopted” and “It was too hard to learn how to participate effectively” are both selected by approximately 10% of respondents.

Taken altogether, this indicates that there is a lot for new participants to learn, not knowing these things adversely affects participation, and this can lead to people feeling excluded.

Email is still universally preferred, but new participants are more comfortable with multiple mechanisms of participation

In Q22 email announcements are the preferred method of being informed of IETF activities by a long way, across all age ranges.  Similarly in Q24 email is the preferred method of participating in IETF discussions and decisions, across all age ranges.  In Q26a the reliance on email is rated as a low hindrance.

However, most of the other participation mechanisms in Q22 and Q24 have a wide spread of preference by age range, with the younger participants comfortable with most of them.  In Q26a younger participants are more hindered by the reliance on email.  When these results are taken together it indicates that the likely trajectory is from email as the dominant mechanism to a range of mechanisms with email the first among equals.