Subject: [newtrk] RE: Interoperability Testing Format

From: Larry Masinter <[email-masked]>
Date: Mon, 08 Aug 2005 10:15:43 -0700

Since the working group encouraged me to continue working on
this idea (at least at the meeting), I thought I would discuss
the idea on the mailing list. Perhaps we could start a new
mailing list to focus on interoperability reports specifically?

After the meeting, I asked people to review existing interoperability
reports and give their opinion about whether they were good
or bad, so we could start working on guidelines for what made
a "good" interoperability report, and understanding what the
evaluation criteria might be.

It was pointed out that: 
 
http://www.ietf.org/internet-drafts/draft-ietf-idr-bgp-implementation-02.txt

might be a "bad example" or at least a difficult case.

Looking at this, the problem seems to be some combination
of formatting, the fact that lots of information is combined
into a single report, and the apparent expectation that 
IETF might publish (as a permanent RFC) what
is really transient information.

What I'm imagining for formalizing implementation reports
would be to take the "schema" and publish *that* as an
IETF consensus document. In this case, it would consist of
the 259 "questions". Perhaps the questions could be formatted
more simply, or put into some XML schema to make them
more mechanically processable? So section 3.20.103 Conflicting
Length of the above internet draft might turn into something
like:

<features>
  <feature id="103" name="Conflicting Length" doc="bgp4" section="6.3">
    If any recognized attribute has 
    Attribute Length that conflicts with the expected length, then 
    the Error Subcode MUST be set to Attribute Length Error
   </feature>
...
</features>
<references>
  <reference id="bgp4">
     .... reference to document identified as [BGP4] in draft...
  </reference>
   ...
</references>

I'd go for a flat structure as a list of features.

The separate implementations "Alcatel", "Cisco",
"Laurel" and "NextHop" would be separate reports, not
explicitly combined into a single report, 
which could be published by their individual authors.

<report>
   <implementation>
       <name>Alcatel blah blah</name>
       <version>... </version>
      ...
   </implementation>
 <status>
     ...
    <feature id="42" value="o">
        All attributes are ordered in 
        ascending order except Extended 
        Community, which is type 16 but we 
        send it out after community 
        attribute.</feature>
    ...
    <feature id="103" value="y" />
   ...
 </status>
</report>

What do you think? Is this on track?

Larry

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 08 Aug 2005 22:24:03 +0200

> evaluation criteria might be.

My absolute favourite implementation report of all time:

<http://www.ietf.org/IESG/Implementations/SMTP-PIPELINE-implementation>

The "more serious-looking" version for Standard, not much longer:

<http://www.ietf.org/IESG/Implementations/SMTP-PIPELINING-Standard-implementation>

Let's make sure we don't REQUIRE anything much longer.

                  Harald

.

---

From: Larry Masinter <[email-masked]>
Date: Mon, 08 Aug 2005 15:06:31 -0700

Harald,

I wouldn't expect "SMTP pipelining" to appear in a separate
ISD or SRD, but rather, for it to be referenced in the
ISD/SRD for "Internet Mail". For that reason, I would also
not want to propose that there be a separate "IRS" 
(Implementation Report Schema) for "SMTP pipelining".

Instead, it looks like it is being reported as a single
feature of existing (already reported) SMTP implementations.

From that point of view, the 'rough consensus' would be
on how to update the feature matrix.

would turn into some kind of

 <update updates="smtp">
    <add>
       <feature id="pipelining">Implements all of RFC xxxx for
           SMTP pipelining, with all options</feature>
    </add>
 </update>

for the schema, and a separate small report file for each
of PMDF, "software.com's post.office", and possibly "MUMPS mail system",
each of which would have some more serious information about the
implementation.

> The "more serious-looking" version for Standard, not much longer:
<http://www.ietf.org/IESG/Implementations/SMTP-PIPELINING-Standard-implement
ation>

would have 7 separate implementation reports.

> Let's make sure we don't REQUIRE anything much longer.

The sum-total of the information would be quite a bit longer, but
the additional length would add to the accountability. The 
problem with the implementation reports you've cited is that
it's hard to know which versions are really being described,
who is doing the reporting, etc.

So 
  (a) yes, there would be an expansion of perhaps 4x in the
   amount of information reported
  (b) the additional information might actually be more useful to
  someone with a new SMTP pipelining implementation who wanted
  to know what to test against.

Larry

.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Aug 2005 16:54:06 +0200

> Let's make sure we don't REQUIRE anything much longer.

I would REQUIRE them to include a date.

     Brian

.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 09 Aug 2005 16:56:37 +0200

> I would REQUIRE them to include a date.

That, and which versions, and who made the assertion.
But not MUCH more.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 11 Aug 2005 06:00:55 -0400

> might be a "bad example" or at least a difficult case.

One problem is that it doesn't satisfy the requirements for advancing
the specification to Draft level; there is apparently only one (rather
than the required two independent) implementation which meets the
specification ("One vendor (NextHop) indicated "O" matching the
specification").

At least the document is arranged in an item-by-item manner which
facilitates identifying specification items which either result in
a failure to have the necessary two implementations or which must be
removed from the specification (BCP 9 sect. 4.1.2 2nd para.) for
advancement.

http://www.ietf.org/IESG/Implementations/rfc-2476-and-draft-gellens-submit-bis-implementation.txt
is a counterexample; it is poorly organized such that it is quite
difficult to determine that there are not in fact two implementations
which fully meet the specification as required by BCP 9.

Worse still is
http://www.ietf.org/IESG/Implementations/RFC2234-implementation-report.txt
which reads like a marketing blurb, contains factual errors, does not
list the specification items required, and lists two implementations,
neither of which meets the specification.

> I'd go for a flat structure as a list of features.

I'd settle for a summary of features something like that provided
at the end of each major section of RFC 1123, though strictly
speaking only the items affecting interoperability (i.e. not
"MAY", "SHOULD", "SHOULD NOT" items) are necessary.  That is, a
brief summary of the specification item and a reference to the
full specification text would be useful.

I'd prefer a formal implementation report submitted as documentation
for advancement to list the two independent implementations which
ostensibly meet all of the requirements, but I could accept Larry's
idea of separate per-implementation reports (however, in that case
it is important that each report contain all of the specification
items in the same order and format so that they can be easily
compared)[1].  Ideally, a matrix listing the specification items as rows
and the implementations as columns would be easiest to examine (are
there two columns that meet all specification items).

---------
1. But not XML.  It's not easily human-readable.  Plain text a la
   RFC 1123 is *greatly* preferable.
.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html