IETF 111 post-meeting survey

The survey results for the IETF 111 post-meeting survey are now available on a web-based interactive dashboard.  We are always very grateful for the detailed feedback that we have received and will continue to process over the next few months.  The commentary below highlights a couple of areas where changes we have made based on feedback have been a success, and areas we still need to work on.  

Analysis

In total 166 responses were received.  Of those 164 participated in IETF 111 from a population of 1329 giving a margin of error of +/- 7.17%. The total numbers of meeting participants has grown (1196 for IETF 110, 1282 for IETF 109) while the survey response rate is significantly down on previous meetings (299 for IETF 110, 258 for IETF 109).

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1.  While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages.

In this commentary a comparison is made with the IETF 110 Meeting Survey results using a comparison of means that assumes the two samples are independent even though they’re not but neither are they fully dependent.  A dependent means calculation may give a different result. A few comparisons are also made using a comparison of proportions.   

Overall satisfaction

The mean satisfaction score for IETF 111 (Q10) was 4.13 with 87% either ‘Satisfied’ or ‘Very satisfied’.  This is a statistically insignificant decline from the 4.20 mean satisfaction score for IETF 110.

Preparedness

Overall, preparation remains good with 83% reporting that they are either sufficiently prepared or well prepared. 4.35% were definitely under-prepared (Q8), well up on the 1.45% for IETF 110, which itself was down from 5% for IETF 109 but still a very small percentage of the total. The comments on this are mixed, with some pointing to the resources for practising with the tools, some bemoaning the lack of in-person meetings and some asking for WG chairs to provide the agenda earlier.

Satisfaction with the agenda

Overall satisfaction with the IETF 111 agenda was at 3.91 (Q12), a statistically insignificant decline from 4.04 for IETF 110 but now rated as 'acceptable' down from 'good'. Looking at the individual parts of the agenda (Q11), most of the satisfaction scores had no statistically significant change except for satisfaction with side meetings, which had a statistically significant improvement, and satisfaction with existing working groups, which had a statistically significant decline. Satisfaction with the opportunities for social interaction remains at 'very poor' and for side meetings, plenary and RGs it rates at 'acceptable'. For all other parts it rates 'good'.

Satisfaction with the meeting structure

Overall satisfaction with the structure of the IETF 111 meeting was 4.08 (Q14), a statistically insignificant drop from 4.20 for IETF 110. However, looking at the individual parts (Q13) there were statistically significant drops in satisfaction for the policy of scheduling online meetings in the timezone of the in-person meeting that they replace, for the overall length of the day and for starting at 12pm "local time", though this latter question was asked slightly differently this time. The first two are both rated as 'acceptable' and the latter is rated as 'poor' and only just above the boundary with 'very poor'. Satisfaction with 9 parallel tracks remains only 'acceptable' at 3.60 but for IETF 109 which has 8 parallel tracks, the score was 3.64 suggesting that a reduction to 8 would not improve satisfaction in any measurable way.

Impact of being fully online

In response to a hypothesis that moving fully online was allowing more people to participate in the IETF than would otherwise have been able to, a new question was added "If this meeting had been a normal (no COVID) in-person meeting in San Francisco with 8+ hour days and free remote participation then which of the follow best applies to you?" (Q15). Only 2 of the 148 who answered this question said that they would not have participated and 5 would have participated in fewer sessions, offset by 4 who would have participated in more sessions, suggesting that the hypothesis does not hold.

Sessions

41% experience no session conflicts (Q18) in IETF 111, up from both IETF 110 and IETF 109 at 35%, though a statistically insignificant increase. Satisfaction at conflict avoidance was 3.76 (Q20), a statistically insignificant increase from IETF 110 at 3.73. This again rates at 'acceptable' and suggests that more work is needed to get it up to 'good'.

For this survey we omitted any questions on sessions running out of time to keep the survey short.

Participation mechanisms

Satisfaction with Meetecho remains 'good' at 4.29 (Q21) and only 'acceptable' for Gather at 3.77 and jabber at 3.68. After experimenting with Gather for four meetings we may need to experiment with alternative mechanisms for social interaction. Satisfaction with the audio streams and youtube videos both experienced statistically significant declines down to 3.84 and 4.09. Further investigation is required to understand this.

Problem reporting

Satisfaction with our response to problem reporting declined from 4.31 for IETF 110 to 4.00 (Q25) (the number of people responding to this question are so small that this is a statistically insignificant decline). From the comments it seems that much of this is down to confusion about what sessions require separate registration and the time it takes to fix that when someone try to join a session that is getting underway. A good improvement suggested in the text (Q26) is to put information on how to get support onto the agenda page.

Final feedback

Clearly, the majority just want to get back to in-person meetings with some using this survey (Q27) to complain about the decision to switch IETF 112 Madrid to an online meeting. Some are concerned that the lack of in-person meetings is making the IETF less relevant and less well supported by community volunteers.