Subject: Mail lost yesterday

From: SM <[email-masked]>
Date: Thu, 29 Aug 2013 15:50:29 -0700

Hello,

Could whoever from the IETF "leadership" who is responsible for the 
matter please comment about the mail loss incident?

I previously commented about the visibility of IETF services.  That 
aspect of the IETF lacks transparency [1] and leaves it to the user 
to guess what is wrong at his/her end.

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/ietf/current/msg68755.html

---

From: Bob Hinden <[email-masked]>
Date: Thu, 29 Aug 2013 16:28:51 -0700

SM,

The place to report IETF operational problems is <[email-masked]>.  I forwarded your email.

Bob

On Aug 29, 2013, at 3:50 PM, SM <[email-masked]> wrote:

> Hello,
> 
> Could whoever from the IETF "leadership" who is responsible for the matter please comment about the mail loss incident?
> 
> I previously commented about the visibility of IETF services.  That aspect of the IETF lacks transparency [1] and leaves it to the user to guess what is wrong at his/her end.
> 
> Regards,
> -sm
>

---

From: SM <[email-masked]>
Date: Thu, 29 Aug 2013 18:29:33 -0700

Hello,

Thanks to Bob Hinden for the quick response.  What follows is a 
general comment.

My message was not a report about an operational problem.  It was 
about the lack of a timely notification, and maybe an explanation if 
anyone deems that appropriate, about these problems to the IETF community.

There is question of whether the services are being monitored.  There 
is also the question of the reliability of the services provided to 
the community.  Usage has likely increased since 2007.  I hope that 
the contract has been updated to take all of that into account.

Regards,
-sm

---

From: Jari Arkko <[email-masked]>
Date: Fri, 30 Aug 2013 11:05:03 +0300

SM:

I certainly agree that in incidents like this, a timely notification is in order. (Of course to the extent that the outage itself allows us to do that. Sometimes the outage or the queue that has built up during the outage delays sending a notification.)

And we normally do send notifications; in this case circumstances (such as people being out sick) conspired to delay sending the notification out. But it did eventually go out, and I'd like to thank Glen, Steve, and the team for dealing with the issue and restoring the service. Based on this experience, we should be able to make notifications faster on future incidents.

Also, I wanted to bring up a point about what to do if you see a problem. Glen already mentioned [email-masked]. This is what you should normally use, if you see a problem. There has been a couple of incidents where this address would not have worked, such as when the entire connectivity to the IETF system is down. Obviously, the IAOC and AMS have worked hard to make sure this does not happen, and we are about to make another update soon that will significantly reduce the risk of site-wide problems. We think it would have prevented yesterday's problem, too. But should you find a situation where the IETF site is unreachable, IAOC and IESG members have a way to alert the staff. Please contact your nearest IESG/IAOC member to bring the issue to their attention. For instance, you can contact me at this e-mail address or [email-masked]/[email-masked] with instant messaging. And since we are spread around the world, we often have noticed the problem already. In yesterday's incident, Pete Resnick noticed the problem and contacted AMS - thanks Pete! 

Jari

---

From: Mary Barnes <[email-masked]>
Date: Fri, 30 Aug 2013 08:17:06 -0500

I will also add that when I requested it Steve sent me a list that
indicated who sent what messages to the mailing lists that I moderate. That
was really helpful as I could ping folks to resend and I was able to resend
those that I had sent myself, so it wasn't too onerous to recover given
that we also had the time window for the lost messages.

Regards,
Mary.

On Fri, Aug 30, 2013 at 3:05 AM, Jari Arkko <[email-masked]> wrote:

> SM:
>
> I certainly agree that in incidents like this, a timely notification is in
> order. (Of course to the extent that the outage itself allows us to do
> that. Sometimes the outage or the queue that has built up during the outage
> delays sending a notification.)
>
> And we normally do send notifications; in this case circumstances (such as
> people being out sick) conspired to delay sending the notification out. But
> it did eventually go out, and I'd like to thank Glen, Steve, and the team
> for dealing with the issue and restoring the service. Based on this
> experience, we should be able to make notifications faster on future
> incidents.
>
> Also, I wanted to bring up a point about what to do if you see a problem.
> Glen already mentioned [email-masked]. This is what you should
> normally use, if you see a problem. There has been a couple of incidents
> where this address would not have worked, such as when the entire
> connectivity to the IETF system is down. Obviously, the IAOC and AMS have
> worked hard to make sure this does not happen, and we are about to make
> another update soon that will significantly reduce the risk of site-wide
> problems. We think it would have prevented yesterday's problem, too. But
> should you find a situation where the IETF site is unreachable, IAOC and
> IESG members have a way to alert the staff. Please contact your nearest
> IESG/IAOC member to bring the issue to their attention. For instance, you
> can contact me at this e-mail address or
> [email-masked]/[email-masked] with instant messaging. And
> since we are spread around the world, we often have noticed the problem
> already. In yesterday's incident, Pete Resnick noticed the problem and
> contacted AMS - thanks Pete!
>
> Jari
>
>

---

From: SM <[email-masked]>
Date: Fri, 30 Aug 2013 07:26:50 -0700

Hi Jari,
At 01:05 30-08-2013, Jari Arkko wrote:
>I certainly agree that in incidents like this, a timely notification 
>is in order. (Of course to the extent that the outage itself allows 
>us to do that. Sometimes the outage or the queue that has built up 
>during the outage delays sending a notification.)

I'll refrain from commenting about the details of the last incident 
as there are extenuating circumstances.

My message was not about the contact address to report operational 
issues.  For what it is worth I did notice an issue with the mail 
archives but I didn't think that it was worth pursuing as my message 
was off-topic.  I read ietf-announce; there wasn't any notification 
there.  I assumed that there was either something wrong on my end or 
it could be some glitch which was not worth the bother.

IASA stated that:

  "The IASA defines the service requirements (statement of work, service
   level agreement)"

That information is not publicly available.

The message you wrote does not say anything about monitoring.  The 
practical question is whether IETF services are being 
monitored.  This is where the IETF "leadership" answers yes and I 
look unreasonable for asking more questions.  On a lighter note I 
have been watching too many IETF movies. :-)

The nit is why is the IETF still using PDT.

Regards,
-sm

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 31 Aug 2013 07:43:57 +1200

On 31/08/2013 02:26, SM wrote:
...
> The nit is why is the IETF still using PDT.

I assure you that things were operationally much worse when the
Secretariat was using EDT.

Really - the service level has improved continuously over the last
eight years. Of course things can always be better, and we learn
collectively from every incident, but compared to various corporate
and campus services I have used, IETF participants have nothing
to complain about.

    Brian

---

From: Yoav Nir <[email-masked]>
Date: Fri, 30 Aug 2013 20:38:34 +0000

> The nit is why is the IETF still using PDT.

Because we don't want to get into a religious war of GMT vs UTC.