IETF 115 post-meeting survey

The results of the IETF 115 London post-meeting survey are now available on a web-based interactive dashboard. Thank you to all of you who respond to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 288 responses, of which 286 participated in IETF 115, 236 onsite and 50 remote. As only 2 of the respondents did not participate in IETF 115 the specific questions for them are not shown in the dashboard, but their views were read and considered. With 1424 recorded participants, this gives the survey a maximum margin of error of +/- 5.18%.

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Satisfaction

Overall satisfaction is 4.28 which is again a good result. With only a few exceptions, the satisfaction scores are the highest they have been in two years and those exceptions are almost all from IETF 113, which appears to have benefitted from a bump in satisfaction due to people being able to meet in-person for the first time in two years. The changes that have had a positive impact on satisfaction include:

• Fine tuning the meeting structure including:
  • starting earlier at 9:30am (thanks to those who noted that I initially had the start time wrong in the survey)
  • scheduling a longer day
  • providing a wider range of session lengths
• Ongoing investment in Meetecho
• Better agenda planning

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above.

Areas for improvement

Gather / Social interaction for remote participants

The satisfaction score is for Gather is Poor at 3.37 and the score for "Opportunities for social interaction" for remote participants was even lower at a Very Poor 2.58. To be entirely open, we know that while Gather is the only opportunity we provide for social interaction for remote participants, it is a poor experience in a hybrid setting, and we are struggling to identify any better alternative, or in fact any alternative worth experimenting with. We have researched physical solutions including telepresence robots and telepresence stands but the scale would only allow a fraction of our remote participants to use them. We have also researched telepresence room technology, but the asymmetry of one location with many participants and many locations with one participant would likely make that a very poor experience.

We are going to continue to try to identify and test alternatives and will consider the feedback in the survey about setting up a long-running MeetEcho room or some mechanism for people to create ad-hoc meetings. In the meantime please contact me directly if you have any further suggestions.

Zulip

Satisfaction for Zulip at 3.73 is similar to that for jabber, the service it replaced, but our goal has to be for it to be a significant improvement over jabber and that to be reflected in the satisfaction scores. For now we are treating this as a familiarity issue and will aim to support people making the switch while expecting results not to improve much for a few more meetings.

Side meetings

The satisfaction score for side meetings at 3.81 is at the high end of this score for the last six meetings, but still notably lower than we would like it. In each post-meeting survey we receive feedback about how to improve side meetings, with those in this survey largely focusing on two issues - the size of the rooms compared to the number of participants, and the issues of side meetings being listed separately from the main agenda. We will look at improvements in these areas, but it should be noted that the IESG has considered this area a number of times and as recently as June 2021 published a blog post setting out its views on the overall approach to side meetings.

Conflict avoidance

Over the last few years we have invested in a new tool to automate the building of an agenda that satisfies any number of pre-defined constraints and we're now aiming to improve the generation of these constraints using the data supplied in the post-meeting surveys. As you might expect, this is a complex process that requires involvement from the IESG, IRTF and WG chairs, which so far this has not moved the needle on the satisfaction score but we are now implementing a more detailed process for considering community feedback. The number of conflicts is of course closely related to the number of sessions per day and the number of parallel tracks and it may be that we have reached the limit of what we can achieve without adjusting either of those.

Number of parallel tracks

There was a bump in satisfaction scores for the number of parallel tracks when we reduced from 9 to 8 but we are still only receiving an Acceptable rating. Any further changes can only be achieved with a major change to the meeting structure, which requires careful consideration.

Plenary

The satisfaction score for the Plenary has hovered around the Acceptable/Good boundary for the last six meetings, despite our attempts to improve it by shortening some parts, increasing the transparency and communicating more in advance so that the Plenary does not become a confrontational session. We would like to improve it to a consistent Good rating but that will will take some time as we need to do more in-depth research with individual community members to understand what they want from the session.

COVID management

We received a total of 32 reports of COVID infection during IETF 115, 13 during the meeting and 19 in the three days following it. It is of course impossible for us to identify the source of any infection, particularly as the meeting took place in an environment where all COVID management policies have been withdrawn and very few people take any personal precautions in public. This is a higher proportion than for IETF 113 or IETF 114 but the community is notably better at reporting to us infections in the immediate aftermath of the meeting and so we may not have good enough data to make an inter-meeting comparison.

Looking at the survey data, I need to start with an apology for a gap in the survey - while we asked onsite participants and those who did not participate at all, their views on the COVID policy, we didn't ask those who participated remotely and so the results will not incorporate views from those who participated remotely because the onsite policy was not to their liking.

While there are multiple comments in the survey from people who strongly disagreed with the masking policy, the data (albeit limited as above) provided a relatively clear result (Q24b) that those views were in a minority:

• 44% would have participated onsite whatever the policy
• 39% lean towards the policy staying as is or being stricter, made up of:
  • 31% might not have participated onsite if the policy had been any looser
  • 8% definitely would not have participated onsite if the policy had been any looser
• 18% lean towards the policy staying as is or being looser, made up of:
  • 12% might not have participated onsite if the policy had been any stricter
  • 6% definitely would not have participated onsite if the policy had been any stricter

When it comes to planning for IETF 116 we are probably going to be constrained by local requirements but this data will be useful for IETF 117 if COVID is still at the level where we need to consider a management policy.

And finally

There are too many individual comments for me to address in detail in this post but please be assured that we will consider those in our planning wherever possible. These include:

• Allowing multiple one-day passes
• Not siting the terminal room in a remote location
• Providing more information about the different WiFi networks we operate
• Making the meeting QR codes easier to find
• Better space management for side meetings

There were a few suggestions that we unfortunately cannot adopt, such as providing meals, but thank you for those anyway.