Subject: RE: [newtrk] RE: Interoperability Testing Format

From: "Robert Snively" <[email-masked]>
Date: Mon, 8 Aug 2005 19:25:13 -0700

Am I being unduly pessimistic in thinking that a set of
implementation reports, each reporting on the 
behavior of a particular product, will not guarantee that
any of the products will actually interoperate with
any other?

I would have expected that I would actually have to test
each pair of devices against each other in the presence of
a monitoring function to observe compliance with the standard
and an error forcing function to guarantee the proper 
behavior in the presence of unusual timing conditions and
errors.  It is not enough to just have one device lie about
its behavior and a second to agree with that lie.

For a complete test report, the test plan, test tools, and
test configuration would have to be defined in one document,
then each additional device brought into the reporting structure
would have to be documented and pass with each of the previously tested
devices.
It is even possible that the fourth or fifth device tested could
identify failures in the standard, the test plan, or one of the
previously tested devices.

Is this something we want to do to progress a document from
proposed standard, through draft standard, to standard?
This would be especially awkward when a later test shows the
standard to still have errors or undocumented side effects.

Is there a solution for this in a three-step (or even a
two-step) process?

Bob

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Larry Masinter
> Sent: Monday, August 08, 2005 3:07 PM
> To: 'Harald Tveit Alvestrand'; [email-masked]
> Cc: 'Bill Fenner'; [email-masked]
> Subject: RE: [newtrk] RE: Interoperability Testing Format
> 
> 
> Harald,
> 
> I wouldn't expect "SMTP pipelining" to appear in a separate
> ISD or SRD, but rather, for it to be referenced in the
> ISD/SRD for "Internet Mail". For that reason, I would also
> not want to propose that there be a separate "IRS" 
> (Implementation Report Schema) for "SMTP pipelining".
> 
> Instead, it looks like it is being reported as a single
> feature of existing (already reported) SMTP implementations.
> 
> From that point of view, the 'rough consensus' would be
> on how to update the feature matrix.
> 
> 
>  
> <http://www.ietf.org/IESG/Implementations/SMTP-PIPELINE-implem
> entation>
> 
> would turn into some kind of
> 
>  <update updates="smtp">
>     <add>
>        <feature id="pipelining">Implements all of RFC xxxx for
>            SMTP pipelining, with all options</feature>
>     </add>
>  </update>
> 
> for the schema, and a separate small report file for each
> of PMDF, "software.com's post.office", and possibly "MUMPS 
> mail system",
> each of which would have some more serious information about the
> implementation.
> 
> > The "more serious-looking" version for Standard, not much longer:
> > 
> >
> <http://www.ietf.org/IESG/Implementations/SMTP-PIPELINING-Stan
dard-implement
ation>

would have 7 separate implementation reports.

> Let's make sure we don't REQUIRE anything much longer.

The sum-total of the information would be quite a bit longer, but
the additional length would add to the accountability. The 
problem with the implementation reports you've cited is that
it's hard to know which versions are really being described,
who is doing the reporting, etc.

So 
  (a) yes, there would be an expansion of perhaps 4x in the
   amount of information reported
  (b) the additional information might actually be more useful to
  someone with a new SMTP pipelining implementation who wanted
  to know what to test against.

Larry

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

.
newtrk resources:_____________________________________________________
web user interface: http://darkwing.uoregon.edu/~llynch/newtrk.html
mhonarc archive: http://darkwing.uoregon.edu/~llynch/newtrk/index.html

---

From: Larry Masinter <[email-masked]>
Date: Mon, 08 Aug 2005 23:24:32 -0700

> any of the products will actually interoperate with

IETF doesn't usually offer "guarantees". But the reports
themselves should allow for you to say more than
"implemented", but also "tested" and possibly the
identity of the other implementations tested against.

I suppose we could even add some mechanism where independent
witnesses to the testing could also assert their observation
of interoperability?

Larry

.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 09 Aug 2005 10:49:07 +0200

> witnesses to the testing could also assert their observation

I kind of like the idea of a slashdot-type forum where the interoperability 
reports appear as "articles", and what people think of them appears as 
"comments".....

.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Aug 2005 16:53:00 +0200

> witnesses to the testing could also assert their observation

I doubt if our lawyer and our liability insurer would be
100% enthusiastic about this. By simply posting reports,
we aren't making any assertions today.

    Brian

.