Subject: Remote Participation Service outage on Wednesday morning

From: Meetecho IETF support <[email-masked]>
Date: Wed, 4 Nov 2015 05:32:34 +0100

Folks,

this morning one of the VMs hosting the IETF Remote Participation 
Service failed, causing a small outage of the service.

Meetecho went down for ~15 minutes around 10.00am. This time frame was 
needed to migrate the impacted sessions and restart them on a working VM.

The outage only affected the sessions running in rooms 301, 302, 303, 304.

The audio-only mp3 streaming was not affected.

We apologize for the inconvenience,
the Meetecho team

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Nov 2015 03:02:29 -0500

Hi.

First, in case anyone doesn't know, at least from Yokohama to
Cambridge, MA, USA, Meetecho announced problems and forced
logging in again three times so far during the plenary.  I just
got an "Internal Server Error" box during Brian Trammel's talk
and then the video froze and audio dropped, so that is probably
a fourth.  These were in addition to the audio going out for
some time during Harald's talk.  After that fourth time, I seem
to be unable to log back in -- two freeze-ups in the login
process and now waiting to connect (little rotating icon for an
extended period.  Finally got another "Internet server error"
and have just given up on Meetecho for the plenary. Not good.

This version of Meetecho with two-way capability also poses a
huge privacy problem.   If I enable active, rather than passive,
participation, I apparently have to agree to give Meetecho
control of both my microphone and webcam in order to get into
the session.  Audio is fine, but I need to be able to mute it
and need onscreen control and verification that it is muted (for
me to have to un-mute _and_ the Meetecho program or controller
to un-mute/authorize me to talk from that end would be fine, but
to have the mic completely under remote control with no
indication to me as to whether it is transmitting or not is
not).    

Video is much worse because, having forced me to enable it if I
may want to make comments, Meetecho insists in turning on my
camera and broadcasting whatever it sees.  That is a nasty
invasion of privacy, especially in the context of participants
in many different time zones.  Based on what one participant
broadcast for a while, apparently before realizing the camera is
on, I'm not the only person who doesn't normally feel a need for
formal dress when participating in a meeting at 3AM my time.

Based on the accumulation of essentially black video frames,
several (or most) remote participants basically put a sock on
the webcam, but that can be hard for laptops with built-in
cameras.  Allowing the Meetecho application to find and enable
the camera is fine, but having it on has got to be under the
control of the user-participant and, IMO, that setting should
start out set off by default.

YMMD, but I consider this so important that I believe the IETF
should insist that Meetecho disable remote live group video
participation in meetings (not remote presentations, assuming
that is a separate function) until this is straightened up.  To
the extent to which the current generation of remote
participation tools is capable of taking over the remote
participant's machine, I also think that IAOC should insist in a
privacy audit of whatever tools are being used before each IETF
meeting.

    john

---

From: Simon Pietro Romano <[email-masked]>
Date: Wed, 4 Nov 2015 09:56:38 +0100

Hello John,

first of all let me apologize for today’s issues. We’re still investigating the situation, but we have most probably identified its root cause.
I’ll try to briefly explain what happened (by avoiding useless technicalities as much as possible) and I hope that this will both clarify the issue and reassure you a  bit with respect to the privacy issues you are raising.
Actually, we misconfigured the virtual queuing server, due to the need to have both moderated participants (the ones who ask for the floor and
subsequently enter the queue) and one un-moderated participant (i.e., a user who is allowed to enter the conversation without being moderated). The latter was needed
for the event we had right before the plenary’s start time. When the plenary started, we failed to reboot the virtual server and, for an incredibly unfortunate combination of events
we got things messed up in the moderation system. This ended up with both an inconsistent behavior and a hanging communication thread between the queuing system and the Meetecho server (which brought the system close to starvation for more than 10 minutes). When such a thread finally crashed, controls were disabled for a while, which resulted in the crazy behavior you mention in your mail. Believe me if I tell you that this should definitely never happen and had never happened before. The virtual queue, though experimental, has proved to work smoothly until today and it did fail at the plenary just because of a misconfiguration form our side.
This said, I personally totally agree with your proposal of defining privacy auditing procedures for all the tools we make use of at the IETF. We are obviously open to not only subject ourselves to this process, but also actively contribute to the design of the entire auditing cycle. 

Hope this helps shed some light on this regrettable episode and believe me if I say that I appreciate your feedback, as usual. I know you’ve been one of the most active remotees so far and
I would like to be sure that you’ll keep on helping us improve our service.

Cheers,

Simon

> On 04/nov/2015, at 09:02, John C Klensin <[email-masked]> wrote:
> 
> Hi.
> 

                     					       _\\|//_
                           				      ( O-O )
   ~~~~~~~~~~~~~~~~~~~~~~o00~~(_)~~00o~~~~~~~~~~~~~~~~~~~~~~~~
                    				Simon Pietro Romano
             				 Universita' di Napoli Federico II
                		     Computer Engineering Department 
	             Phone: +39 081 7683823 -- Fax: +39 081 7683816
                                           e-mail: [email-masked]

		    <<Molti mi dicono che lo scoraggiamento è l'alibi degli 
		    idioti. Ci rifletto un istante; e mi scoraggio>>. Magritte.
               			                     oooO
  ~~~~~~~~~~~~~~~~~~~~~~~(   )~~~ Oooo~~~~~~~~~~~~~~~~~~~~~~~~~
					                 \ (            (   )
			                                  \_)          ) /
                                                                       (_/

---

From: Randall Gellens <[email-masked]>
Date: Wed, 4 Nov 2015 01:04:34 -0800

At 3:02 AM -0500 11/4/15, John C Klensin wrote:

>  the session.

...

>  formal dress when participating in a meeting at 3AM my time.

I'm sure many people working from home would not want their camera 
broadcasting them, and doubtless few other participants would welcome 
the view.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
We shall not cease from exploration / And the end of all our exploring
Will be to arrive where we started / And know the place for the first
    time.   --T. S. Eliot

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Nov 2015 18:12:17 +0900

> broadcasting them, and doubtless few other participants would welcome 

http://www.c-slide.com/

mic is _much_ more difficult

---

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 4 Nov 2015 04:45:06 -0500

>> broadcasting them, and doubtless few other participants would welcome

I have tape over mine and just remove it when needed.

There was an issue where someone in meetecho has entered a Mic:
comment and it was never said at a mic.

I also was disconnected several times from the US, in MA, but it got
better at the end.

-- 

Best regards,
Kathleen

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Nov 2015 05:40:45 -0500

--On Wednesday, November 04, 2015 04:45 -0500 Kathleen Moriarty
<[email-masked]> wrote:

>>> I'm sure many people working from home would not want their
>>> camera broadcasting them, and doubtless few other
>>> participants would welcome the view.
> 

Yeah.  I do something similar as protection against attacks that
could remotely activate the camera, but hope I never have to
treat Meetecho as an attack source.  I also remember an earlier
generation of webcam devices that were equipped with a quite
obvious built-in physical lens cover or shutter arrangement and
regret the passing of that feature

> comment and it was never said at a mic.

It was me; see below.

> I also was disconnected several times from the US, in MA, but
> it got better at the end.

I'm physically not far from you and got the same disconnections
(at least most of which were apparently at the Meetecho end --
see Simon's notes), but simply gave up after circa the third
one, shut the thing down and turned on the IETF audio feed and
logged into the Jabber room with a Jabber client.  So I didn't
experience the "got better at the end" part.  I also don't know
whether, with Meetecho acting up, its Jabber bridge/interface
was working -- most or all of the comments I saw in the Jabber
room after I connected directly seemed to be from people
directly logged into Jabber rather than via Meetecho with your
hum at the end as an exception.

Relative to the question I tried to ask via Jabber, my guess is
that Andrew and Jari were expecting questions to come in via
Meetecho, no one told them it was down or having problems, and
there was therefore no fallback plan such as a designated Jabber
scribe/channel who was logged directly into the Jabber room.   I
tried getting the attention of a few people in the room or on
the stage to tell them about the problem and point folks to
questions in the chat room, but failed.  Stuff happens, Murphy's
Law is still applicable, and it seems to me that one of the key
lessons from this experience is that we should get better at
assuming systems will fail and that we should have detection
mechanisms and fallbacks thought out and in place.  In this
case, the audio feed and slides distributed before the meeting
were adequate, even if not very good, substitutes for Meetecho,
but the failure of the Jabber channel for questions to work was
a much more serious problem.

    john

---

From: Jari Arkko <[email-masked]>
Date: Wed, 4 Nov 2015 21:15:16 +0900

> a much more serious problem.

I’m sorry about this. That was my failure.

Jari

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Nov 2015 10:11:25 -0500

--On Wednesday, November 04, 2015 21:15 +0900 Jari Arkko
<[email-masked]> wrote:

> 
>> Relative to the question I tried to ask via Jabber, my guess
>> is that Andrew and Jari were expecting questions to come in
>> via Meetecho, no one told them it was down or having
>> problems, and there was therefore no fallback plan such as a
>> designated Jabber scribe/channel who was logged directly into
>> the Jabber room.   I tried getting the attention of a few
>...

> I'm sorry about this. That was my failure.

Jari,

Apology accepted but unnecessary.  I am much more concerned
about our learning from the experience and making sure we have
enough backup mechanisms in place to be sure it does not happen
again.

As one part of that, I have suggested to Simon that, if they
discover that their systems are not behaving in top form, they
take responsibility for being sure that whomever is chairing the
meeting or running the in-room mic queue is aware of the
problem.  I suggest to you, Andrew, and WG Chairs that you be
sure that there is a "Jabber scribe" or designee in the meeting
room who is logged directly into Jabber (not via Meetecho or
other intermediary) and alert to system failures and
disconnects.  It also occurs to me that there should be a
contact mechanism for the Secretariat and/or NOC that is not
dependent on the IETF Jabber servers (or, ideally, even the
Secretariat or meeting networks) being up.

None of those things would be necessary if everything could be
guaranteed to work smoothly.  But, when Murphy's Law gets us,
understanding quickly that things aren't working well, being
able to compensate if feasible and to at least be aware of the
problem in real time, are important.

I agree with Randy's unhappiness about what we are giving remote
participants, especially when things don't go completely
smoothly.  We are doing, IMO, lots better than we were a few
years ago, but, as we continue to offer improved capability, we
also continue to raise expectations and dependencies on fairly
complex systems (systems that include the humans as well as the
software).

     best,
       john

---

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 4 Nov 2015 10:31:29 -0500

Sent from my iPhone

> On Nov 4, 2015, at 10:11 AM, John C Klensin <[email-masked]> wrote:
> 
> 
> 

Yes, all of this, plus remote participants should join the jabber room before they connect to meetecho if they want to be sure to have chat capabilities.  It's tougher the other way around and requires you breaking the meetecho session even if some parts are working.  Meetecho sees your name in the chat room and adds a number to the end so you have 2 distinct connections.

Kathleen