Subject: Re: www.ietf.org

From: John C Klensin <[email-masked]>
Date: Mon, 15 Jan 2018 13:56:27 -0500

Hi.

I decided to put this thread aside for the last several days and
have just reviewed it (and the ones is spawned) all at once.
After thinking about the various comments, some observations
(none completely original although my summary may be different
from what others have said in some cases) and a suggestion:

(1) At this stage in the process, [re-]design by committee and
nit-picking are probably not helpful, no matter how interesting
and engaging they might be.   Bug reports to the IETF list
probably aren't either.  On the other hand, to the extent to
which a large collection of 404s or failures on validation or
accessibility tests show up, that goes beyond bugs and reflects
badly on the design and/or implementation processes (for which
see below).

(2) We know there are at least two different audiences for an
IETF web page -- participants trying to get work done and and
people trying to find out what the IETF is about.   Those may
further subdivided: there is a difference between newcomers
interested in better understanding and eventual participation in
the IETF's work, policymakers trying to understand IETF and its
relationship to policy development and work going on elsewhere,
and managers and others trying to make decisions about IETF (or
maybe even ISOC) support, permission of staff members to
participate and travel, etc.  Similarly, there are people trying
to find out what the IETF is doing in a particular topic area
and how they can get involved with it and those whose technical
interest in the IETF is more general or more about the Intere6t
rather than a specific protocol or cluster of protocols.  

I think we all understand that it is impossible to optimize for
all of those audiences and sub-audiences at once, but it is not
clear that the current site reflects that understanding.  For
example, my guess is that the IETF's interactions with IGF (such
as they are) are probably of significant interest to only one of
those subgroups and, for it,  the explanation is probably not
nearly adequate.  I believe that is a symptom, not a particular
problem that, once solved, goes away.

(3) Whatever we do should reflect IETF values about quality,
security, etc.  If FaceOogAppAzonBaba conclude that they should
respond to their audiences and marketing personnel by web sites
with lots of animation, rotating pictures and videos, extensive
use of CSS and JS and links off to other domains, etc., it may
make the practices common and even what lots of people are used
to, but, while we should be aware of that in our design, it
doesn't make those practices "good" (much less "best") or what
we should be doing.   Given the concerns about JavaScript and
the obvious risks of links to non-IETF sites, I think the
designers should be required to consider every instance of
either as a tradeoff and to be ready to explain the importance
of each one to the community, not just to move freely ahead on
each because they think they are pretty and/or effective in
reaching or impressing some audience.  

Given some of the other comments, if there had not been an
earlier  beta and comments on it, and if it were clear that
those comments were reflected in the later version (or the
reasons why not explained if the issues were even vaguely
substantive or strategic), I would not be writing this note.
But it appears to me that the spirit of many of the earlier
comments was not accommodated.

So, assuming two assumptions are correct, a suggestion.   The
first of those assumptions is that the IETF still makes
decisions about policy and strategy (not just narrowly technical
issues) by rough consensus, not by appointing people to make
those decisions for us.  I'm not talking about the details of
web page design, but I am talking about whether security and/or
accessibility is important in our public-facing materials,
whether we want to have a "marketing" stance at all and whether
the "work" pages and goals should be subsidiar6 to it or vice
versa.  If our leadership, or those who whom they have chosen to
delegate authority, don't believe the community is able to
understand those issues and make intelligent tradeoffs about
them, I just hope the Nomcom is listening carefully.   The
second is that there are conditions under which it is
appropriate to say "well, this isn't working" and go back to
design principles and new approaches.  A colleague suggested
50-odd years ago that software modules that demonstrated a large
number of bugs should be treated as deeply defective and
reimplemented from scratch, preferably by different people to
lower the risk of making the same mistakes a second time, after
the goals and design were carefully reviewed.  One has to wonder
whether the process that produced this general approach needs
review and evaluation if, after several iterations, the new site
is still as far from meeting the expectations of the community
as the length and number of comments in these threads would
suggest .. and to wonder whether the collection of bad links
from the new pages, validation failures, etc., are supportive of
the hypothesis that we have the right design and implementation
teams (Ned's recent comments about understanding and recognizing
when work is not good are directly relevant here, as is the
question of whether this is a situation we should be trying to
patch our way out of).

So, can we go back to the design goals and mission given to the
folks who did this, get it posted, and see if there is rough
consensus, now that we have seen what is supposed to be an
implementation according to those goals and see if we still have
(at least rough) consensus about those goals?  Can we ask the
question of how well the implementation matches those goals (and
any other goals that we might add in retrospect?  If the answers
are as negative as I think these threads suggest, can we have an
open discussion about what to do next, rather than another set
of back-room decisions?  And can we be sure that the option of
going with a different team is on the table, if only because the
current team, having spent a lot of time on this effort, is
naturally going to be more inclined to patch than to start
rethinking?  It need not be the choice and, given work done
already, I hope we don't need it, but it seems to me to be
important that it be possible.

best,
   john

---

From: "Adrian Farrel" <[email-masked]>
Date: Mon, 15 Jan 2018 19:17:39 -0000

And fwiw, I put my energies into reporting bugs to [email-masked] as was
requested.

AMS seem to be pretty responsive in fixing the smaller problems (content
related) and passing the larger issues on to "the web developer".

imho, the site was released a couple of weeks early when a little beta test
might have made the transition quieter.

Adrian

jck sed...
> probably aren't either.

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 15 Jan 2018 16:34:21 -0500

On Mon, Jan 15, 2018 at 01:56:27PM -0500, John C Klensin wrote:
> folks who did this, get it posted

It _was_ posted.  And reviewed, and updated.  The SOW is at

    https://iaoc.ietf.org/documents/IETF-Website-SOW-02.pdf

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Randy Bush <[email-masked]>
Date: Tue, 16 Jan 2018 09:39:47 +0900

>     https://iaoc.ietf.org/documents/IETF-Website-SOW-02.pdf

so, clearly that process could have worked a lot better than it seems
to have.  what's to learn to reduce likelihood of a repeat?

randy

---

From: Doug Royer <[email-masked]>
Date: Mon, 15 Jan 2018 18:02:32 -0700

> to have.  what's to learn to reduce likelihood of a repeat?

Suggestion; don't create PDF docs that most people do not load, 
especially on a mobile devices.

-- 

Doug Royer - (http://DougRoyer.US  http://goo.gl/yrxJTu )
[email-masked]
714-989-6135

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 16 Jan 2018 10:36:20 +0000

> to have.  what's to learn to reduce likelihood of a repeat?

For web pages use paste-ups so the reviewer and producer can see what
it will look like.

I would imagine that in the media world paste-ups and story boards are
the primary mode of specification.

- Stewart

---

From: S Moonesamy <[email-masked]>
Date: Tue, 16 Jan 2018 04:53:00 -0800

Hi John,
>from what others have said in some cases) and a suggestion:

In 2009, if I recall correctly, Tony responded to a comment from the 
person managing the "new website" project about it being technically 
impossible to keep old links [1] working.  There was also a 
discussion about Javascript [2] on the new website.

The process [3] followed is the usual for the IETF.  The end product 
is "as is".

IETF policy is usually determined by the IESG.  I am not sure about 
which body determines "IETF values".  Tim Chown pointed out that 
'words like "folks" are very US-centric'.  I didn't receive any reply 
to my question about the Internet Governance Forum blog post even 
though the topic is prominently displayed on the home page.

The Internet is going through a reconfiguration.

Regards,
S. Moonesamy

1. Information about that is available in RFC 7231
2. Javascript is covered in the SOW.
3. The SOW identified three groups as the audience.
4. https://www.ietf.org/mail-archive/web/ietf/current/msg106082.html

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 16 Jan 2018 09:55:01 -0500

> For web pages use paste-ups so the reviewer and producer can see what

That didn't work _either_, since there's been a beta site around for
more than a year.  It appears to have failed in at least three or
maybe four ways:

1.  Some number of people didn't look at it or send any comments, but
nevertheless feel comfortable adding comments now.  I am surprised by
the number of comments that began, "I didn't send any comments during
the beta, but …"

2.  Some number of people believe they sent comments that were not heard.

3.  Some number of people believe that "the leadership" or the project
did not react correctly to comments that were received.

It strikes me also that there could be a fourth issue:

4.  Some number of people are not actually unhappy with the new site
or the approach, but are unhappy with particular content there.

(4) captures the extensive discussion about the photograph from the
IGF meeting, which seemed to be conflated with the site redesign
because, I suspect, people thought that the new photo was a permanent
feature since they were used to an entirely static IETF front page.

I'm actually not sure what to do about any of these.  The first class
I identify above might be told, "You had more than a year to raise
issues; sorry, but we do not have infinite time."

I'm rather more worried about class 2, though I am not sure how many
of those are actually class 3 issues.  If we actually have class 2
problems, it suggests some continued problems in the management of our
infrastructure.  That may not be too surprising given that our
administrative structures generally have been showing some strain for
the last while; so it may be a symptom of a larger problem.

Class 3 issues reflect a failure of consensus on the goals of the
site.  But that is going to be a problem quite likely forever.  In the
case of protocols where we fail to meet consensus, there are two
possible outcomes: the world moves on without us, or the topic in
question really is moribund.  In either case, our lack of consensus
often doesn't matter that much except for the publication of an RFC or
two and whether something is called a "standard".  But for our
administrative activities, the problem is different, because
administrative activities have a progression of their own: simply not
acting is a kind of action too.  This discussion reminds me that the
IETF community is less cohesive than it once was.  This is an
inevitable consequence of an increasing diversity of population, but
it makes consensus on adminstration things much harder.

I think class 4 is just an unfortunate conflation.  There are
questions about the wider cohesion around what we ought to have on the
IETF website, and the disagreement about that is perhaps reflected in
this particular case, but I think if 1-3 were solved 4 would be at
most an annoyance.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Tue, 16 Jan 2018 10:02:54 -0500

(for those who don't like long notes, roughly the last half of
this one is a pair of extended footnotes that can safely be
skipped.)

--On Tuesday, January 16, 2018 09:39 +0900 Randy Bush
<[email-masked]> wrote:

>>> So, can we go back to the design goals and mission given to
>>> the folks who did this, get it posted
>> 

> so, clearly that process could have worked a lot better than
> it seems to have.  what's to learn to reduce likelihood of a
> repeat?

FWIW, I reviewed that SOW carefully after getting Andrew's note
and now remember it and think I even made some comments on it or
its predecessors.   It is clear to me that what we now have is
not consistent with the way I interpreted the SOW when I first
looked at it [1].  I think there are several actionable lessons
implicit in that:

(1) We need to be much more clear about goals and objectives.
Many of the things listed as "objectives" in the SOW are means
to implicit ends.   "Implicit" may not work for us and we may
need to rethink how we structure SOWs.  That is easier said than
done; if it implies skill sets we don't have or don't have
mechanisms to apply, it has possibly-profound implications for
the IASA 2.0 effort.  It also implies that, unless the community
is willing to delegate complete authority to committees that do
not operate in public and transparent ways, actual contractual
terms, not just the SOWs on which the contracts are presumably
based, should be available in the interest of review and
transparency [2].

(2) Especially if a contractor is not already deeply embedded in
the IETF community and culture (and probably if they are), the
language of an SOW or contract needs to be much more clear about
definitions and, if there might be tradeoffs to be made,
priorities.   As just one example, it is now clear to me (and
should have been clear earlier), that, if I say "modernize ...
the website's ... design" to a commercial web site designer, I
am likely to end up with something that looks a lot like a
commercial marketing or user-as-product site, something we
probably did not intend.

(3) No matter how good a job one does of definitions, sometimes
one needs to build models or prototypes in order to be sure all
parties are talking about and expecting the same thing.  That
isn't specific to web sites but is probably a general
engineering design principle.  Stewart's suggestion about
paste-ups and story boards are directly relevant here, but
sometimes working models are needed for people to say "whoops,
you didn't understand what we wanted, time for a reset".
Probably that means that contracts that involve design elements
should be written to specify benchmark points at which mock-ups
or prototypes are presented, with possibility of changes or
course or even cancellation if those monitoring the contract
conclude that the contractor doesn't get it.

That is my answer to "reduce likelihood of a repeat?".  Others
can probably add other things.  As to what we should do not (and
to partially review what I said earlier):

(4)  Treat what is up now, retroactively, as a prototype, being
prepared to write it off as a learning experience (expensive and
too bad, but these things happen).  Go back and review the goals
and objectives in the light of that prototype and decide how it
compares to what we really wanted and adjust accordingly.  As
part of that effort, cut back on the nit-picking or move it
elsewhere.  Then decide whether we can best get to where we want
to end up by adjusting the current implementation, going back
and adjusting the prior one, or starting over.  Then figure out
how to execute on whatever decision we reach, preferably without
another four-year process.

>From experience and observations of other efforts, actually
doing (4) will be hard for this community.  If nothing else,
complaining is easier that careful analysis and specifications
and the design of the IETF web site is probably directly part of
the day job of _very_ few of us.  However, if we can't do it (or
the IAOC can't figure out how to take the steps required), we
probably have the website we deserve whether it is the one we
wanted or not.   And that may be another lesson for the future.

best,
    john

   --------------

These notes can safely be skipped by people who don't like long
notes or details.

[1] Section 2 of the SOW that Andrew cited says the following
(quoted with my comments interspersed):

> 2. Project Goals
> The IETF website redesign will:
> a. Improve ease of navigation,

If navigation by people seeking to do work is included, the
current design is a fail.  I note that it does not say "except
that it is ok to make navigation harder for the more frequent
users of the web site, i.e., the active participate in the IETF".

> b. Modernize and improve the simplicity of the website's
> visual design,

See above about "modern".

>  c. Make the website work better for all classes
> of devices, including smart phones and tablets,

> d. Ensure the website works well for visitors using
> low-bandwidth and highlatency connections,

This is not consistent with large, high-resolution, images, at
least unless the users in those constituencies are constrained
to be very patient.

> e. Adopt responsive design

Whatever that means.   If it means (or includes) "respond to
IETF community requirements, I suggest these threads indicate
that site design is not doing well.

> f. Improve accessibility, and

"Improve" is a little vague, but I think several comments on the
list have suggested that this requirement has not been
well-satisfied either.

> g. Improve the content maintenance processes.

This seems still uncertain to me (only partially because of
"improve"), but the large number of 404s from links off the
deployed home page does not suggest that the sanity checks that
are normally part of good maintenance processes were
incorporated into the implementation.

[2] Yes, I understand that contract negotiation on the IETF list
would be an even worse choice than technical or graphic design
on the IETF list.  I am not suggesting that.  I am suggesting
that this particular result may be an example that suggests that
we have not gotten the balance completely right, at least unless
the IAOC/ IAD can assure us that there are no substantive
provisions in the contract that are not in the SOW, noting that
any provisions for review of intermediate or prototype products,
or for determining whether a delivered product in compliant with
the contract specifications, are, relative to this SOW,
substantive.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 17 Jan 2018 09:47:39 +0100

On Mon, Jan 15, 2018 at 04:34:21PM -0500,
 Andrew Sullivan <[email-masked]> wrote 

>     https://iaoc.ietf.org/documents/IETF-Website-SOW-02.pdf

In that case, it is a problem of implementation, not of
specification. The SOW says that old links must still work (which is
consistent with <https://www.w3.org/Provider/Style/URI>).

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 17 Jan 2018 09:50:04 +0100

On Tue, Jan 16, 2018 at 09:55:01AM -0500,
 Andrew Sullivan <[email-masked]> wrote 

> more than a year.  It appears to have failed in at least three or

I may add a fifth one:

5. There are some aspects of the site that people did not test or
check because they assumed it was only a beta site, after all, and
surely the problems will be fixed with the real site. That's certainly
the case for 404s.