IETF 116 post-meeting survey

The results of the IETF 116 Yokohama post-meeting survey are now available on a web-based interactive dashboard. Thank you to all of you who respond to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 223 responses, of which 222 participated in IETF 116, 163 onsite and 59 remote. As only 1 of the respondents did not participate in IETF 116 the specific questions for them are not shown in the dashboard, but their views were read and considered. With 1740 registered participants, this gives the survey a maximum margin of error of +/- 6.15%. (Note, for previous surveys we have used recorded participants, but due to many people not signing in to sessions, we have switched to registered participants).

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Changes since the last survey

For this survey we added some new questions about accommodation to help us understand what makes a good IETF hotel and to see why people make other choices. The COVID questions were dropped for this survey as it was unlikely these would provide any more actionable data.

Satisfaction

Overall satisfaction is 4.30 which is again a good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above.

Areas for improvement

Gather / Social interaction for remote participants

The satisfaction scores for Gather and "Opportunities for social interaction" for remote participants, continue to be unacceptable. For this meeting, we experimented with a hallway Meetecho session but the feedback is that this was not advertised widely enough. For IETF 117 San Francisco, we will repeat that experiment with better communications around it.

However that works out, it is clear the Gather is not working and we now need to make the difficult decision on whether or not to persist with Gather for in-meeting social interaction (we know that it is successfully used by some Hackathon projects and this would not affect that). Alternatively, it may be that trying to create a video-based opportunity for social interaction is the wrong approach and we need to switch to a chat based opportunity.

Please contact me directly if you have any thoughts on this.

Zulip

Satisfaction for Zulip at 3.77 continues to track that for jabber, the service it replaced, and not show any improvement. While there are some criticisms of Zulip in the survey feedback, the nature of that feedback points to this being primarily an issue of lack of familiarity and only secondarily an issue with the tool and/or how we use it. It is not clear how to address that rapidly given that most of us only use Zulip during IETF meetings and in a limited way. We will continue to look at this and again welcome any thoughts on this.

Side meetings

The satisfaction score for side meetings at 3.73 still shows a relatively high level of dissatisfaction. The feedback on how to improve side meetings is primarily asking for better remote participation facilities. This will be discussed with the IESG before IETF 117 San Francisco.

Conflict avoidance

In preparation for this meeting we implemented a new process to reduce conflicts as described in our blog post using the feedback from these surveys. However, the satisfaction score at 3.94 was basically unchanged and so it may be that we have reached the limit of what can be achieved with 8 parallel tracks. We will try this process again for IETF 117 San Francisco and look to see what impact that has.

Plenary

The satisfaction score for the Plenary continues to hover around the Acceptable/Good boundary. We have not yet been able to do the more in-depth research with individual community members to understand what they want from the session.

Individual comments

The individual comments covered a number of key themes:

• The distance between the venue and the hotel. The feedback is that this was not "one roof" and that this distance was not well advertised. On the first point, RFC 8718 is clear that "one roof" is only a preference and participants should expect there to regularly be meetings in conference centres that are a short walk from the IETF hotel. On the second point, we were unfortunately unable to conduct a site visit until just a few weeks out from the meeting, due to COVID restrictions, at which point we immediately included that information in an update on the meeting, but this was not clear in the subject line. We do not anticipate this happening again, but if it does then we will endeavour to ensure that people are made aware.
• Lack of chairs in the hallways. The feedback is that this prevented a natural flow of conversation. Our strategy so far has primarily focused on providing a Lounge for people move to and hold ad-hoc conversations there, as almost all venues would expect us to hire hallway furniture and have it brought it, set up and then taken down at the end of the meeting. This is clearly not working, particularly in a venue without a bar, and so we will look at bringing in such furniture for future meetings.
• Frustration with masks, with many people stating that they will rethink onsite participation if masks are again mandated. As just announced, masks will no longer be required from IETF 117 San Francisco and beyond.
• In-session attendance record. The feedback is that there were too many people onsite who did not record their attendance in a session by logging in to Meetecho. We agree this is a problem and for the next meeting we will passing around the QR code in each session, and if that does not work we will consider more radical solutions.
• Earlier publication of session agendas, with many explaining that this is very important for them to decide on whether or not they participate in a meeting and some noting that they missed out on sessions as a result or they made an unnecessary trip. This feedback will be passed on to the IESG.

Hotels and accommodation

For the first time, we asked questions about accommodation, partly in response to complaints during the meeting about the distance to the IETF hotel, and partly to inform the ongoing discussion on changes to our venue selection process. The overall satisfaction (Q63) at 4.09 is good.

We asked for views on both the importance of certain factors that affect the choice of accommodation, and satisfaction with those. That enables the production of the table of the importance-satisfaction gap for each factor. This is a technique used to rank the relative "problem" among a set of factors, with the higher the gap, the more of a problem we have with that factor relative to others. We intend to collect this data for more meetings before deciding on how to respond.

And finally

Thank you everyone who responded to this survey, your feedback is much appreciated.