Subject: Re: [newtrk] Key words to Indicate Requirement Levels

From: Brian E Carpenter <[email-masked]>
Date: Sun, 02 May 2004 10:21:00 +0200

Alex Rousskov wrote:
> On Fri, 30 Apr 2004, Larry Masinter wrote:
...
>>Would it be unreasonable to add a NEW set of keywords, for use in
>>requirements documents (which lay out constraints on technical
>>specifications)?
>>
>> CRITICAL there is general agreement that this is a critical
>>          characteristic of any definition ...
>>IMPORTANT most believe that this is an important characteristic ...
>>   USEFUL there is general belief that this is a useful feature...
>>          but that other factors might override;
>>          a definition that does not provide this element is
>>          acceptable.
> 
> 
> Why not use MUST/SHOULD/MAY instead? If properly defined, their
> meaning in a requirements document would match the above.

Because if they mean different things in different types of document,
readers will be even more confused than today. I think Larry's
suggestion is excellent.

    Brian
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Keith Moore <[email-masked]>
Date: Mon, 3 May 2004 09:47:22 -0400

> For keywords in technical specifications:
>
> If we are still going to require "multiple independent
> interoperable implementations of every feature", one way
> to count features is to count each occurrence of
> RFC 2119's MUST, MAY, and SHOULD.

that strikes me as a lousy way to count features.  do we really want to 
constrain specification writers to describe features in MUST, MAY, or 
SHOULD terms?  do we really want to say that a feature doesn't need to 
be tested for interoperability unless it's described in those terms?

> This might focus our discussion of RFC 2119 keywords,
> too: what else, in the IETF process, could they be used
> for, anyway?

seems to me that these keywords should be used sparingly - not 
everywhere they could possibly be used.  in the proper context they are 
quite useful.  over-use would dilute their effectiveness.

> Would it be unreasonable to add a NEW set of keywords,
> for use in requirements documents (which lay out constraints
> on technical specifications)?

it's unreasonable to write requirements documents.  design goals good, 
requirements documents bad.

.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 3 May 2004 09:39:46 -0600 (MDT)

> Because if they mean different things in different types of document,

Why do you assume different meaning? If properly defined, the keywords
meaning would not change based on document type (e.g., the
compliance-based definitions I proposed have the same meaning for
protocols and protocol requirement documents). IMO, before we add more
keywords, we should try to see if 3 old ones can do an equally good
job.

Alex.
.

---

From: Larry Masinter <[email-masked]>
Date: Mon, 03 May 2004 08:55:07 -0700

I agree that blindly counting MUST, MAY, SHOULD keywords is a poor
way of counting 'features', but I still think there's a valuable
relationship between reasonable use of those keywords and the
"multiple independent interoperable implementation of each feature"
requirement and conformance keywords that's worth trying to
spell out.

First, the keywords are best used when they are properties
of implementations; given a set of independent, interoperable
implementations, you should be able to ask -- for each MUST,
MAY and SHOULD -- whether or not the implementation follows
the recommendation.

For the highest level of standard, I would expect multiple
independent implementations that implement every MUST,
that implement every SHOULD or else provide sufficient
documentation for the reasons why they don't. For every
MAY, I would expect to see some cases tested each way,
both with and without the feature.

> it's unreasonable to write requirements documents.  design 
> goals good, requirements documents bad.

I agree that "design goals" is a better name for what the
IETF has traditionally called a "requirements" document.

Since we don't actually require any compliance testing of
"design goals", perhaps it is inappropriate to encourage
use of formal, all-caps keywords. We could still give
guidelines that writers of design goals documents should
indicate the relative importance of the various design
goals, whether they are critical, important, or merely
useful but optional.

Larry
-- 
http://larry.masinter.net

.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 3 May 2004 10:29:49 -0600 (MDT)

On Mon, 3 May 2004, Keith Moore wrote:

> that strikes me as a lousy way to count features.

It is indeed a lousy way, but we are lucky to have at least that in
the "good enough" world. As you know, many specs do not offer even
that level of formalism to identify features.

> do we really want to constrain specification writers to describe
> features in MUST, MAY, or SHOULD terms?

Ideally, specification MUST identify every feature. We are not even
close to that ideal as it requires a great deal of formalism. For the
lack of better identification mechanisms, especially in plain text
world, RFC 2119 keywords are a big help. Specifications SHOULD use
them.

> do we really want to say that a feature doesn't need to

From interoperability testing point of view, the feature does not
exist unless it is more-or-less formally identified and can be
more-or-less tested in isolation. Keywords is one way to more-or-less
formally identify a feature.

> everywhere they could possibly be used.  in the proper context they
> are quite useful.  over-use would dilute their effectiveness.

Specifications should use keywords correctly (e.g., in a context for
which their meaning is defined). "Sparingly" implies that it is better
to avoid keywords, which should not be the point. Given a good
definition, little guess work is required to figure out whether a
keyword should be used.

> it's unreasonable to write requirements documents.  design goals
> good, requirements documents bad.

I agree. However, a design goal often can and should be expressed
using keywords (with polished definitions).

Alex.
.

---

From: Keith Moore <[email-masked]>
Date: Mon, 3 May 2004 13:34:39 -0400

> the "good enough" world.

so if bricks aren't made to fly, are we lucky to have bricks to use
instead of airplanes?

> Ideally, specification MUST identify every feature.

it's not at all clear that this is an ideal.

> For the
> world, RFC 2119 keywords are a big help. 

never mind that they weren't meant for this job, and never mind that
using them in this way is counterproductive to their real purpose.

> Specifications SHOULD use them.

there are plenty of valid reasons to use 2119 keywords sparingly,
and in some cases, to not use them.

> formally identify a feature.

it would not bother me if standards-track specifications defined 
testing criteria at their time of adoption.  I disagree that 
2119 keywords are a good way to do this.

> to avoid keywords, which should not be the point. 

no, sparingly implies that they should be used when appropriate for
their intended purpose, and that authors should not assume that it's
always necessary or appropriate to use them. 

> using keywords (with polished definitions).

strongly disagree.  the point of design goals is really to reach a
shared definition of the problem and of the bounds of the problem,
not to impose formal constraints on the solution.
--
Regime change 2004 - better late than never.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 3 May 2004 12:10:23 -0600 (MDT)

> so if bricks aren't made to fly, are we lucky to have bricks to use

IMO, it would be more accurate to say that we are using raw untreated
bricks that are better than nothing, but our structures are being
damaged by heavy rain and worms penetrate our walls. We would like to
have fire(?) bricks to make reliable buildings. While some might
experiment with airplanes, we cannot even build quality cars yet.

> it's not at all clear that this is an ideal.

There are different people/cultures working within the IETF. They may
never agree what an ideal specification is (beyond trivial stuff like
ID nits).

> using them in this way is counterproductive to their real purpose.

Yes, we often have to use "good enough" tools for the lack of good
ones. As for "counterproductive", not sure what your productivity
measure/optimal is. Using formalisms is indeed often counterproductive
if you want to optimize spec writing speed, but there are other
reasonable measures of productivity (that we will probably never agree
on).

> testing criteria at their time of adoption.  I disagree that 2119
> keywords are a good way to do this.

I agree that they are not good, but IETF does not provide a better way
so we have to use what we've got (unless we have time and support to
build better tools).

> not to impose formal constraints on the solution.

We cannot reach [meaningful/true] shared definition of the
problem/bounds without some level of formalism. Keywords are probably
the only broadly understood formalism provided by IETF and so they are
[mis]used for this purpose. Better keyword definitions or, as some
argue, better keywords, may address the "misuse" part.

> Regime change 2004 - better late than never.

Same for RFC 2119 :-).

Alex.
.

---

From: Keith Moore <[email-masked]>
Date: Mon, 3 May 2004 14:28:11 -0400

> ones.

in a nutshell:

They're not "good enough".  They're not even "good" for this purpose.

> so we have to use what we've got (unless we have time and support to

Before we start insisting that we need tools for defining interop
requirements, why don't we start by trying to actually define interop
requirements in English for standards that we produce?  After we've 
done that for a while, we'll have a much  better idea of what kinds
of tools serve our purposes.
 
> problem/bounds without some level of formalism.  

I've seen too many groups get bogged down in trying to argue
MUST vs. SHOULD on "requirements"  where they didn't even understand
what they were talking about.  I've also seen too many specifications
get submitted for Proposed where 2119 keywords were mis-used or
over-used - for instance, capitalizing every use of "should" even
when the meaning had nothing to do with compliance with the
specification.  I've become firmly convinced that trying to coerce
early problem definitions into "requirements" with MUSTs and SHOULDs
is counterproductive simply because it's trying to nail things down
long before the concepts - or their implications for the protocol
design - are well-understood.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 3 May 2004 13:13:31 -0600 (MDT)

> They're not "good enough".  They're not even "good" for this

Regardless of whether the above assertion is true, I believe it is
possible to change their definitions so that they become noticeably
"better" for many purposes (possibly but arguably including
specification of goals).

> requirements, why don't we start by trying to actually define
> interop requirements in English for standards that we produce?

The 2119 revision would target both goals. That is, it will give
authors ability to define interop requirements and given them tools to
do so without extra work.

Now, we may not agree whether interoperability is actually a primary
goal (small number of buggy implementations often interoperate just
fine), but that is a different question. Whether primary goal is
initial interoperability or long-term compliance, better 2119 would
help.

> After we've done that for a while, we'll have a much better idea of
> what kinds of tools serve our purposes.

I believe we have enough experience to improve 2119. We are not
talking about anything that can be considered state-of-the-art these
days! We are talking about adopting a coherent subset of well-known
and tried spec design principles going beyond ID nits trivia. IMHO,
many IETF specs and their implementations can be significantly
improved if such a set is used and enforced.

> I've seen too many groups get bogged down in trying to argue MUST
> vs. SHOULD on "requirements"  where they didn't even understand what
> they were talking about.  I've also seen too many specifications get
> submitted for Proposed where 2119 keywords were mis-used or

I agree. Any tool, especially a poorly defined one, can be abused or
misused. Better keyword definitions and accompanying rules would help
to solve some of the problems you mention above, but cannot eliminate
any of them. No single action/tool can. That fact should not stop us
from giving good authors and implementors better tools than RFC 2119.

Alex.
.

---

From: Keith Moore <[email-masked]>
Date: Mon, 3 May 2004 15:25:04 -0400

> "better" for many purposes (possibly but arguably including

I don't want to change their definitions.  The ones we have are really
good for their intended purpose.  Why fix something that works?
 
> do so without extra work.

no, it will just make authors' work harder by imposing unrealistic
constraints, and dilute the value of our specifications by making
them harder to understand.

> I believe we have enough experience to improve 2119.

if we really need to improve 2119, let's improve it for its intended
purpose - not dilute its effectiveness by trying to make it serve too
many purposes.

(I can just hear the Golgafrinchians saying, "what if people want fire
that is nasally fitted?")

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 04 May 2004 11:07:28 +0200

Alex,

> keywords, we should try to see if 3 old ones can do an equally good

You keep using the C-word. But our goal isn't compliance; it's
interoperability.

Compliance is a fairly blunt instrument. There were some good anecdotes
about compliance-induced bugs during the OSI years, because OSI took a
formal compliance-based approach (while the Internet was busy interoperating).
I don't want to go there.

The reason I like Larry's definitions above is that phrases like "general
agreement" are a good match for a goals document, and a bad match
for an interoperability specification.

     Brian
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 11:40:23 -0600 (MDT)

On Tue, 4 May 2004, Brian E Carpenter wrote:

> our goal isn't compliance; it's interoperability.

Our goal is interoperability of compliant implementations.
One is useless without the other. I think this is a provable
theorem.

First of all, interoperability claims are only meaningful when the
protocol being implemented is defined. Interoperability claim always
assumes partial or full support of that protocol. In other words,
saying "protocol P implementations I1 and I2 are interoperable" makes
sense only if we know what "P" is. I hope nobody would argue with
that.

Let's assume our goal is interoperability of all implementations of
protocol P, including future P implementations.  Since formal
verifications of P specs is impossible, we can never know for sure
whether we reached that goal. We can only hope that we have.

Our hope is in danger when we find an implementation that is not
interoperable with the other ones. We always have exactly two choices:

	- declare the implementation buggy
	  (i.e., violating P requirements)

	- declare the specification buggy

[ Both declarations can be true at the same time. There are also
out-of-scope implementations but they are treated essentially as
incompliant ones and may require polishing specification scope, so
they do not really add more choices to the above. ]

When we see no new spec bugs for a while, we apply known errata, drop
unused features, and advance the status of P. At this stage our hopes
for reaching "interoperability of all compliant implementations" are
high again.

Note how we have to exclude non-compliant implementations from the
interoperability goal. If we do not, we will never even approach our
goal; we will deadlock with a growing number of "not fully
interoperable" implementations that cannot be changed.

When we declare that a non-interoperating implementation is violating
P requirements, we are making a [non]compliance claim. As the above
process illustrates, it is essential to be able to make non-compliance
claims to stop changing the specification and to know when
specification change is necessary. If we cannot make [non]compliance
claims, we can never even approach our goal.

If our specifications have no [usable] compliance clause, we cannot
make [non]compliance claims. If 2119++ keywords are based on
compliance, we do not have to write compliance clauses (RFC 2119++
would give it to us).

If you compare the utility of compliance-based definitions with the
utility of current keywords based on undefined notions of "absolute
requirement" and "valid reason", you will see that the latter do not
help us to formally reject incompliant implementations simply because
they do not define the relationship between compliance and requirement
violation.

This defect has many side-effects. For example, if we can only say
that "the implementation violates a MUST" we would have much harder
time getting the bug fixed compared to the case where we could say
that "the implementation is not compliant". Folks that make important
development/purchasing decisions usually know about compliance but
often have not read a single RFC. Even if a boss or an implementor has
read RFC 2119, it is still not clear how bad is it to violate a MUST
or a SHOULD.

Compare that with a case of compliance-based definitions where the
effect of violating a MUST or a SHOULD is defined precisely: you
either get a compliant implementation or you do not (and that's the
best we can define the effect in a general-purpose document!
specifications may provide more details in some cases, of course).

> Compliance is a fairly blunt instrument. There were some good
> anecdotes about compliance-induced bugs during the OSI years,
> because OSI took a formal compliance-based approach (while the
> Internet was busy interoperating). I don't want to go there.

Are you sure there are fewer interoperability-induced bugs during the
IETF years?? IMO, it is usually much easier to document, fix, and work
around a bug in the spec than it is to document, fix, and work around
a bug in an implementation you want to interoperate with (which is
broken because it wanted to interoperate with some other
implementation that wanted to interoperate with ...).

> for an interoperability specification.

Larry's definitions rate the "importance" of a feature. Their actual
value for a goals document user is unclear to me because it is unclear
how goal implementations (i.e., candidate protocol specs) would be
compared with each other. Is it better to satisfy (90% of CRITICAL and
90% of IMPORTANT features) than to satisfy (95% if CRITICAL and 10% of
IMPORTANT features)? Will violation of any CRITICAL feature result in
rejection of protocol candidate? If there are now answers to such
questions, then why fool ourselves with an importance rating??

Compliance-based keywords are a good match for both kinds of
documents. Yes, they do not attempt to rate the importance of a
feature. They simply help us write compliant candidate specs and
detect incompliant candidate specs. IMO, their actual utility is at
least on par with importance rating, but they do not pretend to offer
more information than is actually there.

Alex.

P.S. RFC 2119 does not use interoperability to define keywords.
     Polishing RFC 2119 to define keywords in terms of interoperability
     and threats might be a useful exercise, but, again, our users
     understand compliance better than interoperability and/because
     compliance is a much better defined term.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 04 May 2004 13:58:18 -0400

I would put it very differently.

I think that our goal in this regard is (usually, ought to be, ...) that if 
someone makes a good faith effort to implement the specification, there is 
a high probability that such will interoperate with another implementation 
built as a good faith effort to follow the specification.

Such a definition does not lead to an effort to define conformance, but 
rather leads us to focus on making sure that specification is clear about 
what it takes to achieve interoperability.

Yours,
Joel M. Halpern

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 12:04:09 -0600 (MDT)

On Tue, 4 May 2004, Joel M. Halpern wrote:

> Such a definition does not lead to an effort to define conformance,
> but rather leads us to focus on making sure that specification is
> clear about what it takes to achieve interoperability.

Interoperability with what? Compliant implementations!

We have to consciously reduce the scope to compliant implementations
only or we will never approach interoperability no mater how many
interoperability hacks we implement. Interoperability with incompliant
implementations should not be the primary goal, IMO.

Alex.

.

---

From: Keith Moore <[email-masked]>
Date: Tue, 4 May 2004 14:59:04 -0400

> implementations should not be the primary goal, IMO.

Not when stated in those terms.  But in practice, most implementations
are noncompliant in some way - whether due to ambiguity of the
specification, implementor error, failure of the specification to meet
some valid need, or (occasionally) malice on the part of the
implementor.  (yes, this  does happen, far too often).

The point is, it's a bit counterproductive to say "well, implementation
X isn't compliant so it doesn't matter whether we can interoperate with
it". We need to try to write specs that are easy to understand, easy to
implement correctly, meet all of the needs that we can identify, and are
somewhat extensible to meet unanticipated needs.  

And if can do anything to deter malicious vendors from deliberately
causing interop failures in order to give themselves a competitive
advantage, we should strongly consider doing that also.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 04 May 2004 11:55:46 -0700

> One is useless without the other. I think this is a provable

"In theory, there is no difference between theory and practice.
In practice, there is."

Experience with trying to measure "compliance" was so abominably bad in the 
years of the "OSI Wars" that the IETF community long ago decided to drop 
this goal and go for "interoperability" as the measurable goal.

Both are just stepping stones on the way to "useful", which is the 
ultimate, but largely unmeasurable, goal of what we do here (IMHO).

Details only on demand... in a few limited situations, I believe 
"conformance"-like tests serve an useful purpose in a development cycle.
But interoperability is far more important.

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 14:15:26 -0600 (MDT)

On Tue, 4 May 2004, Keith Moore wrote:

> in practice, most implementations are noncompliant in some way -
> whether due to ambiguity of the specification, implementor error,
> failure of the specification to meet some valid need, or
> (occasionally) malice on the part of the implementor.  (yes, this
> does happen, far too often).

Agreed. Note that we usually do not care about overall compliance;
just compliance with respect to the feature(s) that do not seem to
interoperate in a given context.

> The point is, it's a bit counterproductive to say "well,
> implementation X isn't compliant so it doesn't matter whether we can
> interoperate with it".

Agreed. What we should be saying instead is:

	- X is not compliant with regard to feature Foo!
	- Is feature Foo's specification to blame?
	  (If yes, we need to submit an errata, etc.)
	  (If no, let X know)
	- Do we want to interoperate with X?
	  (The answer depends on many things, sometimes on
	  whether X is compliant)
	- If yes, what do we do to interoperate with X?
	...

This is very different from the much more common path:

	- We do not seem to interoperate with X!
	- Let's add a hack here and there so that we interoperate
	  (never mind that it will break 10 future implementations
	  that do the right thing)

> We need to try to write specs that are easy to understand, easy to
> implement correctly, meet all of the needs that we can identify, and
> are somewhat extensible to meet unanticipated needs.

Yes. Both goals are important:

        - all compliant implementations interoperate
        - any good-faith implementation is compliant

and there are others (extensibility, etc.).

> advantage, we should strongly consider doing that also.

IETF can do nothing but give their competitors and 3rd parties tools
to improve compliance. For example, if Foo advertises compliance with
X, Foo has to be sure that their product is indeed compliant or they
can be sued. Today, Foo advertises "support" for X which is not
something that X defines so it cannot be validated.

Alex.

.

---

From: Keith Moore <[email-masked]>
Date: Tue, 4 May 2004 16:31:00 -0400

> to improve compliance. 

not clear.  for instance, if a mail client is expected to submit mail
to a submission server (rather than an ordinary SMTP relay) and the
submission server is expected to check for implementation errors and 
refuse mail that doesn't comply with the spec, then there is at 
least some potential for this to encourage mail client vendors to 
follow the specifications.

---

From: Larry Masinter <[email-masked]>
Date: Tue, 04 May 2004 13:34:05 -0700

I don't think we're getting far trying to deal with the issue
in the abstract. Personally, I'd like to give a little more
attention to the "independent interoperable implementations
of every feature" requirement, since it might shed some
operational light on what we mean by "compliant".

First, there is the question of whether a "feature" can be
implemented in an "implementation" which doesn't implement
all of the features.

For example, with HTTP, there are lots of 'features' in the
form of headers: clients & servers send headers to each other
in the request and response. No implementation that I'm aware
of used ALL of the headers, so of course, we looked, for each
header, of two independent clients and two independent servers
that sent/received the headers (as appropriate for request,
response headers).

Someone on the committee thought this was a "Frankenstein"
approach, that the headers might be used in things that
weren't full implementations. So the implementation itself
might not be "compliant" (in the sense of implementing
every MUST), but it still counted.

Is this OK? Is there a higher level of standard where all
of the implementations documented for interoperability testing
implement all of the MUSTs?

Do the implementations have to be actual implementations of
the protocol?  I had a problem (which was discounted) that many
of the 'implementations' for "file format for internet fax"
weren't implementations of "internet fax". They read & wrote
the file format, but weren't actually part of the overall
processing.

While this is currently acceptable, should it? Could an
implementation of, say, digest authentication in LDAP be
sufficient to advance digest authentication in HTTP?

Perhaps if we could clarify what we would like, in terms of
implementation reports and progression of standards, we might
be better off than talking about "implementation" and
"interoperable" and "compliant" in the abstract.

I'm a little confused about "compliant" vs. "conformant",
since IETF process doesn't currently define either. Are
the words used elsewhere differently?

Larry

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 15:01:26 -0600 (MDT)

On Tue, 4 May 2004, Harald Tveit Alvestrand wrote:

> Experience with trying to measure "compliance" was so abominably bad
> in the years of the "OSI Wars" that the IETF community long ago
> decided to drop this goal and go for "interoperability" as the
> measurable goal.

I did not know interoperability is a measurable goal. What is the
current measurement of, say, HTTP interoperability?

IMO, one cannot meaningfully measure compliance or interoperability.
However, one can detect compliance violations, which is usually all
that is needed as the first step to addressing the problem (with
implementation and/or specification). The only thing that is required
is the definition (a boolean yes/no measure, if you will) of
compliance.

> ultimate, but largely unmeasurable, goal of what we do here (IMHO).

I agree. The problem is that we are dealing with long-term effects
here, but have only short-term knobs. For example, interoperability
among the first few HTTP applications meant pretty much nothing for
the thousands of applications that were later developed, forcing us to
spend unreasonable resources on endlessly patching and hacking what
used to be interoperable (but grossly incompliant) software. Whole new
specifications are now written just to make sure implementations do
not accidently violate basic HTTP MUSTs and screw extensions.

As long as we just keep "measuring" short-term interoperability of
early implementations (instead of quality of long-term specifications)
we will never stop patching.

Alex.
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 15:37:00 -0600 (MDT)

On Tue, 4 May 2004, Larry Masinter wrote:

> implemented in an "implementation" which doesn't implement all of
> the features.

A feature can be implemented correctly in isolation from any optional
features.

> For example, with HTTP, there are lots of 'features' in the form of
> headers: clients & servers send headers to each other in the request
> and response.

HTTP headers are not features! They are means of implementing
features. There is a big difference.

> No implementation that I'm aware of used ALL of the headers, so of
> course, we looked, for each header, of two independent clients and
> two independent servers that sent/received the headers (as
> appropriate for request, response headers).

I hope you went further than just detecting that a header was sent.
With an exception of mandatory headers, it is useless from both
compliance and interoperability point of view to count headers seen.
The only possible use from such an exercise would be to remove unused
headers and methods from the spec.

> every MUST), but it still counted.

How convenient! Why not count tokens or, better, characters instead?
Does this server sends character "C"? Check. With only 256 octets
possible, one would be done in no time!

I wonder whether IESG approved the header-counting method? Or were
they not aware?

> implement all of the MUSTs?

What you described is not a level of standard. It is simply an
indication that some HTTP headers are being used. A level of standard
would be to find two interoperable implementations with regard to each
HTTP feature. Since HTTP features are not well defined, one has to
derive them from hundreds of MUSTs, SHOULDs, and MAYs...

> Do the implementations have to be actual implementations of the
> protocol? I had a problem (which was discounted) that many of the
> 'implementations' for "file format for internet fax" weren't
> implementations of "internet fax". They read & wrote the file
> format, but weren't actually part of the overall processing.

You would not ask this question if File Format for Internet Fax
standard defined compliance subjects. The answer to your question is
standard-specific. A good standard MUST define its scope or compliance
subjects explicitly. If there is no such definition in your case, one
can try to search for MUST/MAY/SHOULDs that apply to "format readers
and writers". This exercise may not yield clear answers as well.

> While this is currently acceptable, should it?

It is unacceptable when standards do not clearly define their
scope/applicability. It should not be difficult to say whether X can
_claim_ compliance with standard Foo (i.e., X is subject to Foo
requirements). This should be the first step when accepting
interoperability reports, for example.

> Could an implementation of, say, digest authentication in LDAP be

It should not be.

> "interoperable" and "compliant" in the abstract.

This sounds like a good idea to me, but I am afraid the number of
people who care about this is very small here. Am I wrong? It also
depends on which progression of standards model we want to use.
Current? Or what has been proposed as its replacement?

> I'm a little confused about "compliant" vs. "conformant", since IETF
> process doesn't currently define either. Are the words used
> elsewhere differently?

Both terms are the same as far as this discussion is concerned.
Different SDOs prefer different versions. For example, W3C likes
conformance. AFAIK, compliance is more common in IETF docs and related
trade publications.

Alex.
.

---

From: Keith Moore <[email-masked]>
Date: Tue, 4 May 2004 17:54:16 -0400

> implement all of the MUSTs?

I think it might be hard to generalize, which is part of why I suggested
that each standard should define its own criteria for how to do interop
tests.  But to answer your question for HTTP, I think that the 
implementations being tested should have to meet all of the MUSTs
before they qualify as candidates for testing optional features.

> the file format, but weren't actually part of the overall

I suspect there was a hidden concern here that by effectively
defining a standard in bits and pieces, there would be no 
interop testing of the whole ball of wax.  The right way to
address that concern (IMHO) is to define a standard for the 
whole ball of wax, rather than to try to impose constraints
on testing the pieces of the standard that make sense in their
own right.

> sufficient to advance digest authentication in HTTP?

no, because there might be different use cases for the two 
that would make one use of digest useful and worthy of
advancement and the other use of digest not useful and
not worthy of advancement.  if you don't have multiple
implementations of digest in HTTP to test there might
be something wrong with digest in HTTP.

Keith
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 16:45:05 -0600 (MDT)

On Tue, 4 May 2004, Keith Moore wrote:

> I think it might be hard to generalize, which is part of why I
> suggested that each standard should define its own criteria for how
> to do interop tests.

Yes! I think this is equivalent (on a meta-level, our disagreements
aside) to the suggestion that standards come with compliance clause.
Many Interoperability or Compliance clause can be derived from RFC
2119++ for specs using that. That's the beauty of this approach: you
get the interop and conformance testing definition for free, just be
cause you use the keywords correctly (except where such definitions
need to be customized).

> But to answer your question for HTTP, I think that the

For the record, there are only two HTTP proxies known to support all
testable MUSTs (RabbIT and DataPower). Since we have not yet tested
clients and servers, I bet there are no MUST-compliant ones except for
perhaps really trivial ones. Thus, if we apply decent quality control
measures to HTTP today, it should be left at PS state. I doubt that
would make many folks happy.

Alex.

.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 04 May 2004 15:10:56 -0700

> current measurement of, say, HTTP interoperability?

<http://news.netcraft.com/archives/web_server_survey.html> - 50 million.

While this may seem totally irrelevant .... the HTTP specification has 
proved quite useful, despite the many abuses of the specification that 
implementations have done over the years.

there are many things that could have been done to make the job of getting 
those 50 million servers interoperable easier - some of which may look a 
lot like "conformance tests". Still, the interoperability is the key thing.

IMHO.

                  Harald

.

---

From: Keith Moore <[email-masked]>
Date: Tue, 4 May 2004 18:56:31 -0400

> aside) to the suggestion that standards come with compliance clause.

compliance != interoperability

I do think there might be a use for compliance tests in rare cases, but 
not as a general rule.

> 2119++ for specs using that.

nope.  there is not anything like a one-to-one mapping between 2119 
keywords and testable features. trying to use 2119 (or a modification 
thereof) in this way just makes the specifications harder to read and 
write.  but we've already been over that.

> That's the beauty of this approach: you

IMHO, that's just wishful thinking and handwaving.  but maybe it's just 
because I haven't seen running code.

what I'd suggest as an alternative: allow working groups to write 
specifications and interop tests using whatever notational tools suit 
them.  nothing requires a standard to use 2119, and occasionally there 
have been good reasons to not do so.  so if there is such a system of 
keywords that can effectively be used for specifying both compliance 
and interop tests, a WG or specification author is free to use it.  if 
it works well, other WGs and authors will follow suit.  to me that 
makes a lot more sense than trying to specify these keywords in newtrk.

> measures to HTTP today, it should be left at PS state.

I don't think so. but it might mean that HTTP needs to recycle at Draft 
and be tweaked to reflect reasonable expectations.

.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 17:03:28 -0600 (MDT)

On Tue, 4 May 2004, Keith Moore wrote:

> specifications and interop tests using whatever notational tools
> suit them.  nothing requires a standard to use 2119, and
> occasionally there have been good reasons to not do so.  so if there
> is such a system of keywords that can effectively be used for
> specifying both compliance and interop tests, a WG or specification
> author is free to use it.  if it works well, other WGs and authors
> will follow suit.  to me that makes a lot more sense than trying to
> specify these keywords in newtrk.

Agreed.

Alex.
.

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 4 May 2004 17:28:26 -0600 (MDT)

On Tue, 4 May 2004, Harald Tveit Alvestrand wrote:

> > What is the current measurement of, say, HTTP interoperability?
>
> While this may seem totally irrelevant ....

Indeed, the relevance of the number of web sites to interoperability
of HTTP implementations is very weak. And we have not even defined
interoperability for HTTP implementations yet.

> the HTTP specification has proved quite useful,

Yes, it has been used a lot. The other adjectives we can use is
complex, erroneous, wasteful. It is difficult to analyse the big
picture impact because we do not have a comprehensive model of the
world.

> there are many things that could have been done to make the job of
> getting those 50 million servers interoperable easier - some of
> which may look a lot like "conformance tests". Still, the
> interoperability is the key thing.

I am not sure how you can objectively say that, given the amount of
known problems and lack of interoperability definition. For example,
if a better specification would let us put the same 50M servers up
with half (50%) the effort, would that mean that current specification
is still something to be proud of, just because it is being used? But
we will never know, of course.

Alex.
.