Subject: [vmeet] Remote Participation Service outage on Wednesday morning

From: Meetecho IETF support <[email-masked]>
Date: Wed, 4 Nov 2015 05:32:34 +0100

Folks,

this morning one of the VMs hosting the IETF Remote Participation 
Service failed, causing a small outage of the service.

Meetecho went down for ~15 minutes around 10.00am. This time frame was 
needed to migrate the impacted sessions and restart them on a working VM.

The outage only affected the sessions running in rooms 301, 302, 303, 304.

The audio-only mp3 streaming was not affected.

We apologize for the inconvenience,
the Meetecho team

---

From: Tony Hansen <[email-masked]>
Date: Tue, 3 Nov 2015 23:43:44 -0500

There was also an outage ~12:55 that affected at least room 502 and
lasted ~10 minutes

    Tony Hansen

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Nov 2015 03:02:29 -0500

Hi.

First, in case anyone doesn't know, at least from Yokohama to
Cambridge, MA, USA, Meetecho announced problems and forced
logging in again three times so far during the plenary.  I just
got an "Internal Server Error" box during Brian Trammel's talk
and then the video froze and audio dropped, so that is probably
a fourth.  These were in addition to the audio going out for
some time during Harald's talk.  After that fourth time, I seem
to be unable to log back in -- two freeze-ups in the login
process and now waiting to connect (little rotating icon for an
extended period.  Finally got another "Internet server error"
and have just given up on Meetecho for the plenary. Not good.

This version of Meetecho with two-way capability also poses a
huge privacy problem.   If I enable active, rather than passive,
participation, I apparently have to agree to give Meetecho
control of both my microphone and webcam in order to get into
the session.  Audio is fine, but I need to be able to mute it
and need onscreen control and verification that it is muted (for
me to have to un-mute _and_ the Meetecho program or controller
to un-mute/authorize me to talk from that end would be fine, but
to have the mic completely under remote control with no
indication to me as to whether it is transmitting or not is
not).    

Video is much worse because, having forced me to enable it if I
may want to make comments, Meetecho insists in turning on my
camera and broadcasting whatever it sees.  That is a nasty
invasion of privacy, especially in the context of participants
in many different time zones.  Based on what one participant
broadcast for a while, apparently before realizing the camera is
on, I'm not the only person who doesn't normally feel a need for
formal dress when participating in a meeting at 3AM my time.

Based on the accumulation of essentially black video frames,
several (or most) remote participants basically put a sock on
the webcam, but that can be hard for laptops with built-in
cameras.  Allowing the Meetecho application to find and enable
the camera is fine, but having it on has got to be under the
control of the user-participant and, IMO, that setting should
start out set off by default.

YMMD, but I consider this so important that I believe the IETF
should insist that Meetecho disable remote live group video
participation in meetings (not remote presentations, assuming
that is a separate function) until this is straightened up.  To
the extent to which the current generation of remote
participation tools is capable of taking over the remote
participant's machine, I also think that IAOC should insist in a
privacy audit of whatever tools are being used before each IETF
meeting.

    john

---

From: Simon Pietro Romano <[email-masked]>
Date: Wed, 4 Nov 2015 09:56:38 +0100

Hello John,

first of all let me apologize for today’s issues. We’re still investigating the situation, but we have most probably identified its root cause.
I’ll try to briefly explain what happened (by avoiding useless technicalities as much as possible) and I hope that this will both clarify the issue and reassure you a  bit with respect to the privacy issues you are raising.
Actually, we misconfigured the virtual queuing server, due to the need to have both moderated participants (the ones who ask for the floor and
subsequently enter the queue) and one un-moderated participant (i.e., a user who is allowed to enter the conversation without being moderated). The latter was needed
for the event we had right before the plenary’s start time. When the plenary started, we failed to reboot the virtual server and, for an incredibly unfortunate combination of events
we got things messed up in the moderation system. This ended up with both an inconsistent behavior and a hanging communication thread between the queuing system and the Meetecho server (which brought the system close to starvation for more than 10 minutes). When such a thread finally crashed, controls were disabled for a while, which resulted in the crazy behavior you mention in your mail. Believe me if I tell you that this should definitely never happen and had never happened before. The virtual queue, though experimental, has proved to work smoothly until today and it did fail at the plenary just because of a misconfiguration form our side.
This said, I personally totally agree with your proposal of defining privacy auditing procedures for all the tools we make use of at the IETF. We are obviously open to not only subject ourselves to this process, but also actively contribute to the design of the entire auditing cycle. 

Hope this helps shed some light on this regrettable episode and believe me if I say that I appreciate your feedback, as usual. I know you’ve been one of the most active remotees so far and
I would like to be sure that you’ll keep on helping us improve our service.

Cheers,

Simon

> On 04/nov/2015, at 09:02, John C Klensin <[email-masked]> wrote:
> 
> Hi.
> 

                     					       _\\|//_
                           				      ( O-O )
   ~~~~~~~~~~~~~~~~~~~~~~o00~~(_)~~00o~~~~~~~~~~~~~~~~~~~~~~~~
                    				Simon Pietro Romano
             				 Universita' di Napoli Federico II
                		     Computer Engineering Department 
	             Phone: +39 081 7683823 -- Fax: +39 081 7683816
                                           e-mail: [email-masked]

		    <<Molti mi dicono che lo scoraggiamento è l'alibi degli 
		    idioti. Ci rifletto un istante; e mi scoraggio>>. Magritte.
               			                     oooO
  ~~~~~~~~~~~~~~~~~~~~~~~(   )~~~ Oooo~~~~~~~~~~~~~~~~~~~~~~~~~
					                 \ (            (   )
			                                  \_)          ) /
                                                                       (_/