Filter by topic and date
IETF 119 post-meeting survey
- Jay Daley IETF Executive Director
12 Apr 2024
IETF 119 Brisbane was held 16-22 March 2024
The results of the IETF 119 Brisbane post-meeting survey are now available on a web-based interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.
Analysis
We received 255 responses, 254 of whom participated in IETF 119, 190 onsite and 64 remote. With 1206 registered participants, this gives the survey a maximum margin of error of +/- 5.47%.
The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.
Question changes since the last survey
For this survey we added a new question about the block on submitting I-Ds two-weeks before a meeting, and some new questions about time spent in Australia at the request of the local tourism agencies that offered us funding for this meeting. In order to fit in those questions we dropped the matrix questions about each individual part of the agenda and structure of the meeting and side meetings.
Actions taken following the last survey
For this meeting, we made the following changes, prompted by survey feedback:
- The registration desk now opened on the Saturday and we no longer had a separate registration for Hackathon participants.
- The onsite badge was changed to card stock for a lower environmental impact and redesigned for better visibility.
- We implemented a new shared workspace, replacing the separate lounge and terminal room.
Satisfaction
Overall satisfaction is 4.24, down on recent meetings but still a good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.
The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)
| IETF 119 Brisbane | IETF 118 Prague | IETF 117 San Francisco | IETF 116 Yokohama | IETF 115 London | IETF 114 Phila. | |
|---|---|---|---|---|---|---|
| Overall satisfaction | 4.24 🟢 | 4.42 🟢 | 4.30 🟢 | 4.30 🟢 | 4.28 🟢 | 4.19 🟢 |
| AGENDA | ||||||
| Overall agenda | 4.05 🟢 | 4.20 🟢 | 4.16 🟢 | 4.18 🟢 | 4.22 🟢 | 4.06 🟢 |
| Sessions for new WGs | - | - | 4.19 🟢 | 4.17 🟢 | 4.12 🟢 | 4.15 🟢 |
| Sessions for existing WGs | - | - | 4.22 🟢 | 4.22 🟢 | 4.22 🟢 | 4.10 🟢 |
| Sessions for WGs | - | 4.32 🟢 | - | - | - | - |
| BOFs | - | 4.12 🟢 | 3.95 🟡 | 4.11 🟢 | 4.10 🟢 | 4.09 🟢 |
| Sessions for existing RGs | - | - | 4.12 🟢 | 4.14 🟢 | 4.10 🟢 | 3.95 🟡 |
| Sessions for RGs | - | 4.09 🟢 | - | - | - | - |
| Plenary | - | 3.94 🟡 | 3.99 🟡 | 3.98 🟡 | 3.98 🟡 | 3.98 🟡 |
| Side meetings | - | 3.79 🟡 | 3.75 🟡 | 3.73 🟡 | 3.81 🟡 | 3.73 🟡 |
| Hackathon | - | 4.27 🟢 | 4.25 🟢 | 4.34 🟢 | 4.35 🟢 | 4.30 🟢 |
| HotRFC | - | 3.90 🟡 | 3.89 🟡 | 3.84 🟡 | 4.21 🟢 | 3.94 🟡 |
| Pecha Kucha | - | 4.00 🟢 | 4.15 🟢 | - | - | - |
| Office hours | - | 4.07 🟢 | 3.98 🟡 | 4.23 🟢 | 4.00 🟢 | 4.09 🟢 |
| Opportunities for social interaction | - | 4.06 🟢 | 4.11 🟢 | 3.72 🟡 | 3.98 🟡 | 3.89 🟡 |
| STRUCTURE | ||||||
| Overall meeting structure | 4.15 🟢 | 4.34 🟢 | 4.28 🟢 | 4.28 🟢 | 4.28 🟢 | 4.19 🟢 |
| Start time | - | 4.49 🟢 (9:30am) | 4.28 🟢 (9:30am) | 4.16 🟢 (9:30am) | 4.28 🟢 (9:30am) | 4.20 🟢 (10:00am) |
| Finish at 5pm Friday | - | 3.65 🟡 | - | - | - | - |
| Length of day | - | 4.20 🟢 | 4.30 🟢 | 4.30 🟢 | 4.32 🟢 | 4.10 🟢 |
| Number of days | - | 4.18 🟢 (5+2) | 4.27 🟢 (5+2) | 4.30 🟢 (5+2) | 4.32 🟢 (5+2) | 4.30 🟢 (5+2) |
| Session lengths | - | 4.38 🟢 (60 / 90 / 120) | 4.41 🟢 (60 / 90 / 120) | 4.36 🟢 (60 / 90 / 120) | 4.32 🟢 (60 / 90 / 120) | 4.25 🟢 (60/120) |
| Break lengths | - | 4.38 🟢 (30/90) | 4.32 🟢 (30/90) | 4.38 🟢 (30/90) | 4.36 🟢 (30/90) | 4.25 🟢 (30/90) |
| Number of parallel tracks | - | 3.94 🟡 (8) | 4.08 🟢 (8) | 4.01 🟢 (8) | 3.90 🟡 (8) | 3.86 🟡 (8) |
| PARTICIPATION MECHANISMS | ||||||
| Meetecho | 4.50 🔵 | 4.47 🟢 | 4.35 🟢 | 4.45 🟢 | 4.45 🟢 | 4.23 🟢 |
| Gather | - | - | 3.52 🟡 | 3.46 🔴 | 3.37 🔴 | 3.06 🔴 |
| Zulip | 3.86 🟡 | 3.94 🟡 | 3.66 🟡 | 3.77 🟡 | 3.73 🟡 | 3.56 🟡 |
| Jabber | - | - | - | - | - | - |
| Audio streams | 4.27 🟢 | 4.27 🟢 | 4.02 🟢 | 4.21 🟢 | 4.04 🟢 | 4.05 🟢 |
| YouTube streams | 4.24 🟢 | 4.36 🟢 | 4.32 🟢 | 4.36 🟢 | 4.25 🟢 | 4.22 🟢 |
| CONFLICTS | ||||||
| Conflict avoidance | 3.88 🟡 | 4.00 🟢 | 3.90 🟡 | 3.94 🟡 | 3.91 🟡 | 3.78 🟡 |
| VENUE & ACCOMM | ||||||
| Overall accommodation | - | 4.34 🟢 | 4.07 🟢 | 4.09 🟢 | - | - |
| Overall venue | - | - | 3.90 🟡 | - | - | - |
| Location | - | 4.52 🔵 | 3.60 🟡 | - | - | - |
| Venue facilities | - | 4.43 🟢 | 4.07 🟢 | - | - | - |
| Cost of rooms | - | 3.54 🟡 | 2.87 ⚫️ | - | - | - |
| Availability of rooms | - | 4.32 🟢 | 4.07 🟢 | - | - | - |
| ONSITE | ||||||
| Overall | 4.37 🟢 | 4.50 🔵 | 4.29 🟢 | - | - | - |
| Badge collection | 4.71 🔵 | 4.70 🔵 | 4.69 🔵 | - | - | - |
| WiFi | 4.09 🟢 | 4.17 🟢 | 3.98 🟡 | 4.06 🟢 | 4.10 🟢 | 3.82 🟡 |
| QR Codes | 4.31 🟢 | 4.18 🟢 | 4.11 🟢 | - | - | - |
| Break F&B | 3.75 🟡 | 4.05 🟢 | 4.44 🟢 | - | - | - |
| Breakout seating | 3.93 🟡 | 3.80 🟡 | 4.08 🟢 | - | - | - |
| Signage | 3.96 🟡 | 4.15 🟢 | 4.22 🟢 | - | - | - |
| Coffee carts | 4.08 🟢 | 4.64 🔵 | 4.56 🔵 | - | - | - |
| Childcare | 3.76 🟡 | 4.12 🟢 | 4.06 🟢 | - | - | - |
| Welcome reception | 3.97 🟡 | 4.08 🟢 | - | - | - | - |
| Farewell reception | 4.15 🟢 | 4.34 🟢 | - | - | - | - |
Special questions
Pre-meeting block on I-D submission
In this survey, a special question was asked "Internet-Drafts cannot be submitted or updated in the two weeks before the meeting (unless an AD approves an exception). Please rate how much you agree with the following:". With the caveat that the following results only reflect the views of 250 people while the total number of active IETF participants is over 7,000, the results of this questions indicate strong support for the current practice and either leaving it as is, or allowing WG chairs to also approve exceptions.
The full results are detailed in the table below:
| Coding Mean | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | |
|---|---|---|---|---|---|---|
| This rule is helpful for my preparation | 3.89 | 2.67% | 7.11% | 20.00% | 39.11% | 31.11% |
| This rule should stay as it is | 3.81 | 4.95% | 5.41% | 22.07% | 39.19% | 28.38% |
| This rule should stay but WG chairs should also be allowed to approve exceptions | 3.33 | 9.05% | 12.67% | 27.15% | 38.46% | 12.67% |
| This rule should be extended to other methods for updating I-Ds, such as GitHub merges | 2.89 | 15.84% | 21.29% | 32.18% | 19.31% | 11.39% |
| This rule should stay but for a shorter period than two weeks | 2.50 | 18.18% | 33.64% | 30.91% | 14.55% | 2.73% |
| This rule should be removed entirely | 2.01 | 42.72% | 27.70% | 20.66% | 3.76% | 5.16% |
Why people participated remotely
For the third time, we asked people who participated remotely (Q5), why they did and if they would have preferred to participate onsite (Q5a). Once again, the major factor, cited by 75% of people, was the lack funding to travel.
Funding
We asked the question "What was the primary source of funding for your onsite participation?", the results of which were:
- "My employer" 75.14% (133 respondents)
- "A grant or bursary" 9.04% (16 respondents)
- "My own business" 5.65% (10 respondents)
- "My personal funds" 5.08% (9 respondents)
- "One or more clients" 5.08% (9 respondents)
- "Other (please specify) - no respondents selected this option
Good results
Meetecho
The satisfaction score for Meetecho continues to slowly move up, though not in a statistically significant way, and for the first time Meetecho was rated as Excellent at 4.50.
Registration desk / badge collection process
This again achieved an exceptionally high score at 4.71.
Areas for improvement
Signage
This scored lower than previous meetings and analysis of the comments seems to indicate that the reason for this is because this was a very large space and therefore much harder to navigate than normal, which our signage did not adequately cover. We also made some changes to the signage, which we need to review.
Side meetings
While an overall satisfaction score for side meetings was unintentionally omitted, the satisfactions scores for individual elements (Q71) remain low. There are also multiple comments (Q16) about problems with side meetings.
Conflict avoidance
Again, satisfaction scores with this remain low and further work is needed to devise a new strategy to improve this.
Breakout seating
While there was plenty of breakout seating at this venue, it was generally a bit of a walk to get to it, indicating that our previous analysis is correct that participants want this immediately outside of meeting rooms.
Break food and beverage
We intentionally provided less food in breaks in an effort to tackle the extensive food waste we have had a previous meetings, and that combined with a significant shortfall in meeting sponsorship funds, meant that only limited breakfast catering was provided. Together these resulted in a lower satisfaction score. Feedback from the comments seems to indicate that we need to provide less sweet snack food and more healthy options.
Barista coffee
Yes, sorry.
Individual comments
The individual comments (Q9, Q16, Q27) cover a number of key themes, many of which have already been covered above:
- Brisbane. Comments here are very mixed with some commenting that this was an excellent venue and it was about time that we went back to Australia, with others complaining that it was both too far and a difficult timezone for remote participation.
- I-Ds and Agenda. There are multiple comments about how the timing of I-Ds and agenda affect effective preparation. Most of these ask for better adherence to the rules, while a few ask for more flexibility.
- WiFi. There are multiple comments identifying issues people had with the WiFi during the meeting.
- ALLDISPATCH. There are multiple comments about this, some in favour and some against, all of which will be passed to the IESG.
And finally
Thank you everyone who responded to this survey, your feedback is much appreciated.