Subject: Re: Last Call: draft-ietf-opsawg-operations-and-management (Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions) to BCP

From: Sam Hartman <[email-masked]>
Date: Wed, 03 Jun 2009 15:22:30 -0400

I was assigned this document as a secdir review.
I'm not sure I have any comments in that capacity.

However, I do have significant comments on the last call of this document.

I appreciate the work that has gone into this document.  People have
worked hard to find examples, cases and even pithy
sayings/architectural principles from many areas of the IETF.  The
document tries to be broad and to look at a lot of options.  I think
that it would be reasonable to publish this document as the current
thinking of the ops WG or ops area.  It may even be reasonable to use
something like this as guidelines for network/routing/sub-network
layer protocols to think about management and operations.
However this document is not suitable for publication as a BCP because:

* It does not reflect practices across significant areas of the IETF
* It does not provide clear, actionable guidelines
* It is not sufficiently clear to be understood outside the O&M area.

My background.  I have never formally been involved in the O&M area.
However, I have studied SNMP as a participant and AD for the ISMS Wg.
I have studied syslog as a user, operator, and as the AD who sponsored
many of syslog's base documents.  I have studied netconf as an AD who
reviewed the spec.  I've worked as a small enterprise operator.  I'm a
chair of a WG (LISP) that needs to consider operations and management
issues.  I believe that I should be able to read and understand this
document; I believe that I'm well within its target audience.

I got as far as the bottom of page 18 before giving up.

Here are details on my concerns based on what part of the document
I've read.  I'm happy to invest significant time to get other
reviewers to evaluate whether I have valid concerns; if I get others
to read the document and consider whether I have a point, then I've
done what I set out to do.

1) It does not reflect practices across significant areas of the IETF

This is a concern that the document does not have broad enough
consensus to be a BCP.  I believe that significant areas of the IETF
do not view management interoperability as a goal--much less a guiding
principle of management.  I've been involved in discussions in the
Kerberos working group where we explicitly discussed this and came to
the conclusion that management interoperability was not something
anyone in the room was going to work on.  We did work on an
information model which covers aspects where people believed some degree of management interoperability would be desirable.  It does not cover monitoring, faults, or the like--only provisioning of the database.

Similarly, I'm quite certain that most web server vendors, ATOM
implementors, etc do not want management interoperability.  I
understand that ISP operators very much do want management
interoperability.  I think that we have a significant conflict here
and I think that we cannot approve such a BCP without addressing that
conflict.  One possible resolution would be to find an subsection of
the IETF that actually agrees with these guidelines and scoping the
BCP.

Similarly, it has been my experience that the desire to standardize
information model semantics is not universal across the IETF.

My recommendation for determining if I have a valid concern here is to
get one or two WG chairs from each area to read this document and
comment explicitly on whether the approach to management and
managability outlined in this document is consistent with practice in
their area and WG.

Far too much of the text was specific to management of network
elements such as routers, switches and the like.  The apps and
security area use LDAP for a lot of things that I think would count as
management in this spec.  I don't know how to separate control plane
and data plane issues on my web server.  Enough of the text seemed
specific to network management instead of service/application
management that I would find applying this document in such a context
more frustrating than helpful.  This specific sub-concern could be
entirely addressed by properly scoping the applicability of the
document.

 2) It does not provide clear, actionable guidelines

I was looking at this document and trying to figure out what it would
mean for my WG if this document was approved.  As best I can tell it
would simply add justifications for discusses that would be hard to
debate fairly.  Do I have to write up an information model for my
protocol?  If the ops AD says I do, what grounds do we use to
determine whether that's reasonable?  The answer may be "yes and you
don't get to disagree," but I can't tell from the document.  If it's
going to be a BCP, that needs to be clear to me as a WG chair.

For what it's worth, I don't think we have the necessary consensus to
require people to write up information models and would not be part of
such a consensus at this time.  If this is a list of things I might
want to consider then why is it a BCP?

If we want to come up with a set of things that need to be documented
when appropriate, then I could support that.  I think we need to
clearly distinguish the normative process requirements from the set of
things to consider in such a case.  The set of things someone might
interpret as normative in this spec is far too broad.

  I do agree that
some of the concerns I am raising in this section could be leveled
against BCPs we've approved in the past--even BCPs that I sponsored in
at least one case.  I don't think that diminishes the concerns: we try
and learn from our mistakes.

3) It is not sufficiently clear to be understood outside the O&M area.

This document does an excellent job of clearing up a few o&m concepts:
the difference between information model, data model, modeling
language and protocol.

The document indicates that several distinctions are important, such
as the distinction between operations and management, the distinction
between configuration and other forms of management, etc.  The
configuration vs non-configuration management distinction seems very
important because I believe there is a growing belief that the set of
management protocols you use depends on what you are managing.  I hope
no one plans to use syslog to configure their routers--perhaps the
only worse choice would be IPFIX.

Unfortunately, while I agree that these distinctions are important, I
don't think the document succeeds in making them.  I have reread
section 2 and the first part of section 3 a couple of times.  I still
don't understand the difference between operations and management;
section two talks about management a lot and section 3 talks about
my operations model.  I don't understand what an operations model is,
even after reading the text a couple of times.

I found section 3 very frustrating.  I'd be reading along thinking I
was talking about management for one purpose, thinking about how that
interacted with protocols for that purpose, and then suddenly
something gets thrown in that completely fails to fit.  For example,
I'd be thinking about how to manage configuration state and then
suddenly I'm being told to define the semantics of my counters
consistently.  Both of those are fine things to discuss, but not in
such a way that a reader gets them mixed together.  The mental context
swap is too jarring and I found completely lost me.

In conclusion, I'm sorry that I could not be more constructive.  I
really a lot has gone into this document.  I realize that the goals
are important.  However I don't think we're particularly close to done
at least if the target is a BCP.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 04 Jun 2009 08:15:26 +0200

Sam,

Thanks for posting your review.  It caused me to have a look at the 
document, because I've been considering writing something of a missive 
on my own.  I have to say that I enjoyed reading this draft up to about 
Section 3, and then I came to some problems.  I'll spell those out in a 
separate message, but I agree with you on one key point: this document 
does not does not provide actionable guidance, and actually avoids the 
use of 2119 language.  Clearly that was a design goal.  But I think in 
the end it has led to a good INFORMATIONAL document, or even the 
beginnings of a good text book, and not necessarily a BCP.

Also, I agree with you that interoperability is not the first and 
primary goal.  Clearly, and I hope Dave and others will agree, the 
primary goal is the ability to roll out new useful functions and 
services in a way in which they can be managed in a scalable manner, 
where one has understood the network impact (or total cost, if you will) 
for that service.  And Section 2 provides a really nice illustration of 
that, with regard to what happened at Berkeley.

I have another problem with 3.2: I think the SMI and other 
registry-based forms of structured data may be overrated in some cases.  
Sometimes what is being managed is very simple data, through a central 
service.  One has to question who the consumer of the data is prior to 
determining how one encodes that data, or even what data is truly 
necessary.  Consider the case of an application that is simply calling 
home to determine if there are patches.  How much structured data does 
that service need?  In this context, 3.2 is often irrelevant to the 
application running on some host, but it may be very relevant to the 
central system that is doing the updating.

What I would suggest is that some additional context is necessary 
throughout the document to indicate really who we are talking to.  Are 
we limiting our role to router management?  What happened to the toaster 
with an SNMP stack?  Would we use SNMP today for that toaster?  Who 
would manage it?  Well, let's put it in a more serious context and swap 
toaster for major appliance like air conditioner or refrigerator or PDU.

Ok, and again.  Thanks Dave and others for writing the doc.

Eliot

On 6/3/09 9:22 PM, Sam Hartman wrote:
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>
>

---

From: SM <[email-masked]>
Date: Wed, 03 Jun 2009 22:22:23 -0700

>However this document is not suitable for publication as a BCP because:

The document is a good piece of work.  The considerations discussed 
in it may help other areas understand the thinking of the ops area as 
management and operations considerations are often 
overlooked.  Bernard Aboba posted some interesting comments on 
Section 1 of this draft for the Last Call.

I don't think that the document scales well from an IETF perspective 
as a BCP.  As such, I do not support its publication as a BCP.

Regards,
-sm

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 4 Jun 2009 11:16:05 +0200

Sam,

Thank you for your review and opinions. 

I would like to remind you and let many people that are not aware about
the history of the document know one fact that may be important. This
document is an outcome of the discussions hold at the IESG retreat in
May 2006. I was then the 'fresh' AD bringing this issue to the IESG
table, we discussed approaches on dealing with management in the IETF
and the need for a different approach of looking at management than the
'write a MIB' which was the rule in the IETF WGs until then. I took the
action item to 'write a draft' on this issue - which then developped in
this piece of work chartered in the OPSAWG. 

...

> 
> This is a concern that the document does not have broad 
> enough consensus to be a BCP.  I believe that significant 
> areas of the IETF do not view management interoperability as 
> a goal--much less a guiding principle of management.  I've 
> been involved in discussions in the Kerberos working group 
> where we explicitly discussed this and came to the conclusion 
> that management interoperability was not something anyone in 
> the room was going to work on.  We did work on an information 
> model which covers aspects where people believed some degree 
> of management interoperability would be desirable.  It does 
> not cover monitoring, faults, or the like--only provisioning 
> of the database.
> 
> 
> Similarly, I'm quite certain that most web server vendors, 
> ATOM implementors, etc do not want management 
> interoperability.  I understand that ISP operators very much 
> do want management interoperability.  I think that we have a 
> significant conflict here and I think that we cannot approve 
> such a BCP without addressing that conflict.  One possible 
> resolution would be to find an subsection of the IETF that 
> actually agrees with these guidelines and scoping the BCP.
> 
> Similarly, it has been my experience that the desire to 
> standardize information model semantics is not universal 
> across the IETF.

I believe that the document recognizes the variance in approaches for
the different areas, protocols, and working groups in the IETF and for
this reason rightly avoids making a prescription or imposing a fixed
solution or format in dealing with operational considerations and
manageability aspects of the IETF protocols. I think that it does make
however the point that operational deployment and manageability aspects
need to be taken into considerations for any new IETF work. The
awareness of these issue should exist in any work the IETF engages with,
after all we develop technologies and protocols to be deployed and
operated in the real life Internet, not abstract mathematical models. It
is fine if a WG decides that its protocol needs not interoperable
management or no standardized data model, but this should be the result
of discussions and decisions, not of mission. 

Dan

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 4 Jun 2009 10:58:14 +0100

In the discussion of IETF consensus of this document and its position as a 
BCP or otherwise, can I throw into the melting pot 
draft-ietf-pce-manageability-requirements-06.txt

This particular I-D was developed within the PCE working group to apply only 
in that working group. It covers similar topics, but is more focused on 
ensuring that the protocols that are developed in PCE are manageable.

The I-D has rough WG consensus (or did at the time I stopped being chair a 
few months ago), but is not mandatory to implement within the WG.

It is my opinion that those PCE I-Ds that have taken the draft on board have 
produced better solutions that will be more successfully implemented and 
deployed.

Cheers,
Adrian

---

From: Sam Hartman <[email-masked]>
Date: Thu, 04 Jun 2009 07:22:56 -0400

>>>>> "Dan" == Romascanu, Dan (Dan) <[email-masked]> writes:

    Dan> Sam, Thank you for your review and opinions.

    Dan> I would like to remind you and let many people that are not
    Dan> aware about the history of the document know one fact that
    Dan> may be important. This document is an outcome of the
    Dan> discussions hold at the IESG retreat in May 2006. I was then
    Dan> the 'fresh' AD bringing this issue to the IESG table, we
    Dan> discussed approaches on dealing with management in the IETF
    Dan> and the need for a different approach of looking at
    Dan> management than the 'write a MIB' which was the rule in the
    Dan> IETF WGs until then. I took the action item to 'write a
    Dan> draft' on this issue - which then developped in this piece of
    Dan> work chartered in the OPSAWG.
I certainly appreciate the work that has gone into this draft.  I'm
not sure why the origins here are important.  If you're saying that it
should have special status because the original discussion happened at
the IESG level, I disagree.  If you're saying that the content has
broad consensus because it started at the IESG level, I disagree.  If
you're saying that it's important work with a long history, I agree.

    Dan> I believe that the document recognizes the variance in
    Dan> approaches for the different areas, protocols, and working
    Dan> groups in the IETF 

I strongly disagree that it succeeds in this goal.
I agree it tries.
As an example, section 3.1 says that the primary goal when considering management should be interoperability.
That's a broad statement that does not have IETF consensus and is inappropriate for a BCP.

    Dan> and for this reason rightly avoids making
    Dan> a prescription or imposing a fixed solution or format in
    Dan> dealing with operational considerations and manageability
    Dan> aspects of the IETF protocols. I think that it does make
    Dan> however the point that operational deployment and
    Dan> manageability aspects need to be taken into considerations
    Dan> for any new IETF work. The awareness of these issue should
    Dan> exist in any work the IETF engages with, after all we develop
    Dan> technologies and protocols to be deployed and operated in the
    Dan> real life Internet, not abstract mathematical models. It is
    Dan> fine if a WG decides that its protocol needs not
    Dan> interoperable management or no standardized data model, but
    Dan> this should be the result of discussions and decisions, not
    Dan> of mission.

It's not at all clear to me from this document that would be fine.
That's one of my most serious problems with the document.

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 4 Jun 2009 13:35:06 +0200

Hi Sam,

A clarification and a clarification question in-line.

Dan

> -----Original Message-----
> From: Sam Hartman [mailto:[email-masked]] 
> Sent: Thursday, June 04, 2009 2:23 PM
> To: Romascanu, Dan (Dan)
> Cc: Sam Hartman; [email-masked]; [email-masked]
> Subject: Re: Last Call: 
> draft-ietf-opsawg-operations-and-management(Guidelines for 
> Considering Operations and Management of NewProtocols and 
> Protocol Extensions) to BCP
> 
> I certainly appreciate the work that has gone into this 
> draft.  I'm not sure why the origins here are important.  If 
> you're saying that it should have special status because the 
> original discussion happened at the IESG level, I disagree.  
> If you're saying that the content has broad consensus because 
> it started at the IESG level, I disagree.  If you're saying 
> that it's important work with a long history, I agree.

None of these - just background information to place this document in
context. 

...

> That's one of my most serious problems with the document.

Can you clarify? I cannot understand what is clear to you and what is
not, and with which statement you do not agree.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 04 Jun 2009 16:18:40 +0200

[this one more publicly]

Dan,

Based on the goals you set out below, I would argue that the document is 
too long.  I would recommend sticking with principles and calling out a 
few examples.  I think this is done reasonably well in Section 2, and 
less so elsewhere.  I  would also suggest that you scope the document, 
because what you write below wasn't clear to me on first read.

Eliot

---

From: Eric Rosen <[email-masked]>
Date: Thu, 04 Jun 2009 14:05:31 -0400

Adopting this document  as a BCP would  be a serious mistake, and  I hope it
will be strongly opposed.

There is absolutely no evidence that following the dictates of this document
will improve the quality of the  IETF's work, and we certainly know it won't
improve the timeliness.  There is no evidence that ignoring it will harm the
Internet.

I don't see that OPSAWG has  any business imposing requirements on work done
by  other WGs  in other  Areas.   There are  already an  adequate number  of
obstacles to getting work done.  The  last thing we need is another required
"considerations" section, or another set of excuses for ADs from one Area to
block documents from other Areas.

---

From: "Randy Presuhn" <[email-masked]>
Date: Thu, 4 Jun 2009 12:00:29 -0700

Hi -

> From: "Eliot Lear" <[email-masked]>
> To: "Sam Hartman" <[email-masked]>
> Cc: <[email-masked]>; <[email-masked]>
> Sent: Wednesday, June 03, 2009 11:15 PM
> Subject: Re: [OPSAWG] Last Call: draft-ietf-opsawg-operations-and-management (Guidelines for Considering Operations and Management
of New Protocols and Protocol Extensions) to BCP
...
> that, with regard to what happened at Berkeley.

Unfortunately, I think the IETF has only adopted the first sentence
of this paragraph.  The hard work (and it is hard work) to build
interoperable *management* seems to have largely been abandoned
in the IETF.  I think the state of the netconf work attests to the
shift to just shoveling around vendor-specific blobs, rather than
management information.

> registry-based forms of structured data may be overrated in some cases.

They're not a panacea.  The SMI has weaknesses that can make
models cumbersome.  I haven't seen a data representation or way
of representing data semantics that was ideal for all purposes, and
I doubt I ever will.  Unfortunately, this recognition that nothing is perfect
seems to have led the IETF dangerously close to "anything goes."
One would hope that this document would help WGs avoid the worst
excesses.

> that service need?

If there will ever be more than one patch, if there are dependencies between
patches, if there is every the need to revert, if there are hardware or model
dependencies....  It's really easy to over-simplify, and end up needing a
whole bundle of band-aids as the system's needs evolve (or are recognized).

...
> toaster for major appliance like air conditioner or refrigerator or PDU.

The IETF management discussions have increasingly limited themselves
to what it takes to manage a few types of network gear.  It seems to me
that the IETF abandoned any hope of doing system management, or even
management of multi-vendor systems, a long time ago.  This may be a bug,
this may be a feature.

The thrust of this document seems to me to be a recognition of this
state of affairs, and that consequently, if we're going to let a thousand
vendor- or application-specific flowers bloom, a little advice to help
avoid some of the mistakes of the past would be helpful.

Consequently, I think having this document is a good thing.  The advice
it gives reflects the current fragmented state of affairs with respect to
management information, and would be useful to folks trying to do
a good job in this environment.  I can support this as an informational
document.  I could somewhat reluctantly go along with making it BCP -
our experience with that catch-all designation makes it clear that "Best"
only means "what we could more-or-less agree on" and "Current"
can mean "we really hope something better will come along, but aren't
going to hold our breath."

Randy

---

From: Sam Hartman <[email-masked]>
Date: Thu, 04 Jun 2009 15:47:02 -0400

>>>>> "Eric" == Eric Rosen <[email-masked]> writes:

    Eric> I don't see that OPSAWG has any business imposing
    Eric> requirements on work done by other WGs in other Areas.

Obviously I agree with this statement.  However I do believe that the
ops area can propose and build consensus on requirements that we all
agree to follow.  I think this document may be a starting point in
such a discussion.  I think Eric and I probably both agree it should
not be the final word.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 04 Jun 2009 15:54:07 -0400

To put it differently, the OPS area has as much right to propose their 
requirements as any other area (Transport Congestion, Security, ...) 
has.  And generally, the community has listened to such requests and 
gone along with them.

Yes, we have produced a bit of a problem that our initial standards now 
have a quality bar comparable with completed work.  But we shouldn't 
suddenly pick on OPS for that.  If we are going to address that problem, 
it ought to be in a coherent fashion that discusses how we deal with all 
the legitimate requirements, including the fact that stuff has to be 
operable.

This does not mean we have to simply accept what they (OSP) say.  But it 
does mean we should give it a fair review, looking at the details, 
rather than objecting on principle.

Yours,
Joel M. Halpern

---

From: Eric Rosen <[email-masked]>
Date: Thu, 04 Jun 2009 16:17:17 -0400

> This does not mean we have to simply accept what they (OPS) say.  But it 

This is  absolute nonsense.  Most of  the people actually doing  work in the
various areas do not have the  time, interest, or expertise to do a detailed
review of  an OPS document.   However, these are  the people who are  in the
best position to determine whether "OAM Considerations" would help or hinder
the work that they do.

If we are going to talk about adding new hoops for folks to jump through, we
should first  discuss whether any such  hoops are necessary.   We should not
start the  discussion by looking at  the details of  the particular proposed
hoops. 

> the OPS area has as much  right to propose their requirements as any other
> area  (Transport  Congestion, Security,  ...)   has.   And generally,  the
> community has listened to such requests and gone along with them.

Generally,  the community (i.e.,  the folks  doing the  work in  the various
areas) has never even heard  about these proposed requirements until after a
BCP  appears, at  which  time they  are  told that  the  BCP "has  community
consensus".  Perhaps  you're familiar with Douglas  Adams' "The Hitchhiker's
Guide to the  Galaxy".  To paraphrase, "but the plan  for the destruction of
earth was clearly posted in  the planning department on alpha centauri, it's
not our fault you didn't see it".

---

From: Andy Bierman <[email-masked]>
Date: Thu, 04 Jun 2009 13:38:28 -0700

> the legitimate requirements, including the fact that stuff has to be 

I agree, and I also agree with Randy about the lack of
standardized NETCONF content.  It is a clear indication
that the vendors do not want any standardized configuration
content.  Every time 'content' has come up for charter consideration,
the consensus has been to work on something else instead.

But the NETCONF/YANG pieces are coming together, and it will
allow WGs much more power and flexibility than SNMP/SMIv2
to define management interfaces which fit better with
the native CLI than anything the IETF has had before.

That is about as far as OPS-NM area can go.
If a protocol WG do not want to define a
standard configuration interface, then it
does not matter how well SNMP or NETCONF supports
this sort of work.

(One might question whether using the same NM standardization
methodology for 20 years and achieving consistently pathetic
results means the methodology itself needs to be changed,
not just the technology.)

> rather than objecting on principle.

Andy

---

From: Sam Hartman <[email-masked]>
Date: Thu, 04 Jun 2009 16:54:06 -0400

>>>>> "Eric" == Eric Rosen <[email-masked]> writes:
    Eric> If we are going to talk about adding new hoops for folks to
    Eric> jump through, we should first discuss whether any such hoops
    Eric> are necessary.  We should not start the discussion by
    Eric> looking at the details of the particular proposed hoops.

Agreed.  I think that if we don't have agreement with the
goals/high-level of this type of proposal, it's completely reasonable
to block review of details until that reaches agreement.

---

From: "Tom.Petch" <[email-masked]>
Date: Fri, 5 Jun 2009 09:33:03 +0200

----- Original Message -----
From: "Adrian Farrel" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, June 04, 2009 11:58 AM

> draft-ietf-pce-manageability-requirements-06.txt

Yes, a clear concise set of steps to follow when writing an I-D.

The other comment I would make on the I-D under Last Call is on the first
sentence of the abstract, to whit,
"  New protocols or protocol extensions are best designed with due
   consideration of functionality needed to operate and manage the
   protocols. "

True, but not, I think, addressed by this document.  There is plenty in this I-D
for the I-D editor but little for the protocol designer. The latter might
benefit
from advice on how to structure elements of the PDUs and the interchange of
PDUs; it may be difficult to know whether or not a network is functioning if
there is no keepalive.  (A comparable discussion surfaced recently on
apps-discuss over the best way to encode lengths).

I appreciate that this I-D does not set out to give such advice but think that
someone reading the abstract might expect it to.

Tom Petch

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 5 Jun 2009 20:44:55 +0200

> not our fault you didn't see it".

There is a defined IETF review process and we just go through it and
from what I can tell the process works in making people comment on the
draft that was proposed to become a BCP.

While comments on the content of the draft are fine, I find comments
on the process pretty much out of scope.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>