Subject: Re: Conformance Test for HTTP 1.1

From: "Carl Kugler/Boulder/IBM" <[email-masked]>
Date: Mon, 9 Oct 2000 13:39:51 -0600

>
>I just want to put my two cents into this conversation:
>
>I think the idea of doing compliancy testing is great.  And the idea of
>having one "check everything test" is also a good thought.  However, how
do
>we guarantee that the test scenarios created are actually following the
>"specs"?
>

I was thinking along the lines of a script (or script fragment) for each
MUST in the spec.  MUSTs are supposed to be verifiable, right?  All
compliant implementations, regardless of manufacturer/developer, must do
the MUSTs, right?   Using scripts makes it easy for people to inspect a
script and correct it if it isn't according to spec.

>I think this is something better left to outside agencies to address.  The
>testing game tends to get to be too industry biased.  Whether intentionally
>or not you will see tests similar to this proposed one done and get totally
>different results depending on who does it.
>
>I know this actually sounds like a good argument to create a "standard
>test", but in my opinion this leads the doorway too wide open to start
>skewing the tests in favor of one manufacturer/developer vs. another one.  I
>realize that there are currently many industry leaders involved in this
>organization and they provide valuable insights.  However, they are just
>involved in the CREATION of standards, not in judging the conformance to
>them.
>
>In short, while this is a good idea with the best interests of everyone in
>mind, I think this is probably stepping outside of the charter of the
>organization.
>
>-kh
>
>----- Original Message -----
>From: "Mark Nottingham" <[email-masked]>
>To: "Miles Sabin" <[email-masked]>
>Cc: <[email-masked]>
>Sent: Friday, October 06, 2000 11:30 AM
>Subject: Re: Conformance Test for HTTP 1.1
>
>
>>
>>
>> I think proxies are the biggest target, because they're so hard to
>implement
>> correctly, and so much more complex. In my experience, there's a fairly
>wide
>> variance in how implementors choose to interpret the spec.
>>
>> Of course, once you do one for proxies, it's relatively easy to get client
>> and server test suites out of it.
>>
>>
>>
>> On Fri, Oct 06, 2000 at 10:24:14AM +0100, Miles Sabin wrote:
>> > Mark Nottingham wrote,
>> > > I've lately been considering starting discussion of
>> > > development of something within the W3C, as it was involved
>> > > in the development of the HTTP, and has an established
>> > > history of developing similar tools (although I'm not sure if
>> > > W3C can formally commit resources).
>> > >
>> > > If anyone has any thoughts about this, please share them,
>> > > because I'd like to get this moving.
>> >
>> > This sounds like a fine idea (tho', as you say, it's an open
>> > question whether or not the W3C would be able to commit
>> > resources).
>> >
>> > Do you have any particular emphasis in mind: server, clients,
>> > or proxies, or all equal weight on all?
>> >
>> > Cheers,
>> >
>> >
>> > Miles
>> >
>> > --
>> > Miles Sabin                       Cromwell Media
>> > Internet Systems Architect        5/6 Glenthorne Mews
>> > +44 (0)20 8817 4030               London, W6 0LJ, England
>> > [email-masked]          http://www.cromwellmedia.com/
>> >
>>
>> --
>> Mark Nottingham
>> http://www.mnot.net/
>>
>>
>

---

From: "Caveman" <[email-masked]>
Date: Tue, 10 Oct 2000 09:23:10 -0500

Carl,

Once we start doing any kind of compliancy checking we face the proverbial
"slippery slope".  What comes next?  Seperate tests for things that MAY be
done according to the specs?  Things that SHOULD be?

I think the best thing to do is stay out of the compliancy checking business
all together.

Thanks,

Keith

----- Original Message -----
From: "Carl Kugler/Boulder/IBM" <[email-masked]>
To: "Caveman" <[email-masked]>
Cc: "Mark Nottingham" <[email-masked]>; "Miles Sabin"
<[email-masked]>; <[email-masked]>
Sent: Monday, October 09, 2000 2:39 PM
Subject: Re: Conformance Test for HTTP 1.1

---

From: "Scott Lawrence" <[email-masked]>
Date: Tue, 10 Oct 2000 11:02:37 -0400

> done according to the specs?  Things that SHOULD be?

I don't understand the problem you seem to see.  

Slippery slope to what?  Interoperability?

--
Scott Lawrence         Architect             <[email-masked]>
Virata             http://www.virata.com/         http://emweb.com/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 10 Oct 2000 11:55:07 -0700

Err, could you give a more solid demonstration as to why this is not good?

-- 
Mark Nottingham
http://www.mnot.net/

---

From: "Keith Hoffman" <[email-masked]>
Date: Tue, 10 Oct 2000 12:04:09 -0500

The simplest answer as to why this isn't good is that it's outside the
charter of the IETF.  This organization is here to create standards.  Not to
validate/judge whether someone is compliant with them.

----- Original Message -----
From: "Mark Nottingham" <[email-masked]>
To: "Caveman" <[email-masked]>
Cc: "Carl Kugler/Boulder/IBM" <[email-masked]>; "Miles Sabin"
<[email-masked]>; <[email-masked]>
Sent: Tuesday, October 10, 2000 1:55 PM
Subject: Re: Conformance Test for HTTP 1.1

> > > >skewing the tests in favor of one manufacturer/developer vs. another
one.
> > I
> > > >In short, while this is a good idea with the best interests of
everyone
> > in

---

From: Balachander Krishnamurthy <[email-masked]>
Date: Tue, 10 Oct 2000 13:18:44 -0400

>charter of the IETF.  This organization is here to create standards. 
>Not to validate/judge whether someone is compliant with them.

compliancy to a protocol is one of the key expectation of a specification.
the folks who worked on the spec have a clear interest in seeing whether
all the effort they went to in terms of M/S/M were met. it is not an issue
of "judging" and saying "oh sorry, ee.software is broken".

claiming that ietf should not be interested in compliancy appears a bit 
odd to me. we all stand to gain from making sure that implementations
are compliant to a protocol spec. ietf should be interested. at least 
they were interested last year when they invited me to give a talk at 
a plenary sesion last year during their d.c. meeting - the talk was
on http/1.1 protocol compliancy.

what is wrong with ietf-sponsored group coming up with a test suite 
for an open protocol specification? anyone who wants have their
software tested against it can do so. ietf is unlikely to be partial
in the construction of a compliancy test suite. 

cheers,
bala

balachander krishnamurthy
http://www.research.att.com/~bala/papers

---

From: "Scott Lawrence" <[email-masked]>
Date: Tue, 10 Oct 2000 13:20:56 -0400

> validate/judge whether someone is compliant with them.

Correct.  But that doesn't mean that it wouldn't be valuable both to
implementors and to the IETF process to have a test suite exist
somewhere.  Even if the development of that suite wouldn't be an
appropriate IETF WG (which I agree it would not), one of the
weaknesses of the current IETF process is that there is not enough
feedback regarding which parts of the standards really get done.  I
helped run the effort to document what features had been done for the
transition from PS to DS for HTTP/1.1, and that data was pretty poor
quality.

On the subject of SHOULDs - many of them in the HTTP/1.1 specs would
probably have been MUSTs but for some backward compatibility problems
that doing so would have introduced.  Over time, it will be of benefit
to be able to actually measure whether those have been implemented as
the authors of the specs intended.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 10 Oct 2000 13:25:19 -0700

Right. That was pointed out earlier; discussion was as to where it would be
appropriate. It wasn't clear if your mail was focused on just the HTTP-wg
(which is dormant, and about to close anyway), or on the larger picture.

-- 
Mark Nottingham
http://www.mnot.net/