Subject: Re: [tap] Wakey wakey

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 16:39:16 +0100 (CET)

Hi, folks!

Time to wake this baby again, now that the Perl QA hackathon in Vienna 
is closing in. :)

About the XML version of the document:

   http://github.com/gaurav/test-anything-protocol/blob/master/draft-vaidya-test-anything-protocol-00.xml

Are we considering this one the authoritative version now? (Just to be 
clear.)

Gaurav Vaidya said:
>
> Our bigger issues are: we're still missing an "IANA considerations" 
> section (do we need it? The JSON RFC has one, but perhaps we can get 
> by without an explicit 'application/tap' MIME type for now?). We 
> also have atleast the CRLF/LF and encoding to sort out, but adapting 
> the wording from the XML specification doesn't seem too hard.

I think having an established MIME type for TAP streams would be very 
useful, and definitely worth jumping through some hoops for.

>>> and start circulating it through the larger Perl community for 
>>> feedback?
>
> Would it be easier to share the link to the wiki page instead of the 
> RFC? That way, contributors could easily make the changes or record 
> comments directly at testanything.org.

I think so, but it would be nice to be clear on which text is the 
"real" one (authority-wise.)

- Salve

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <[email-masked]>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)

---

From: Ovid <[email-masked]>
Date: Tue, 2 Mar 2010 07:54:51 -0800 (PST)

--- On Tue, 2/3/10, Salve J Nilsen <[email-masked]> wrote:

> From: Salve J Nilsen <[email-masked]>

> Time to wake this baby again, now that the Perl QA
> hackathon in Vienna is closing in. :)
> 
> Are we considering this one the authoritative version now?
> (Just to be clear.)

Why are '#' characters not allowed in descriptions?

  my $builder = Test::Builder->new;
  $builder->todo_start('hash mark filtering not implemented')
  unlike $result, qr/#/, 'Result does not contain #';
  $builder->todo_end;

Sometimes that's something may want in a description and it reads naturally.  I'd like to see the hash mark (octothorpe, whatever), escaped:

  ok 3 - Result does not contain \# # TODO hash mark filtering unimplemented

> Gaurav Vaidya said:
> > 
> > Our bigger issues are: we're still missing an "IANA
> considerations" section (do we need it? The JSON RFC has
> one, but perhaps we can get by without an explicit
> 'application/tap' MIME type for now?). We also have atleast
> the CRLF/LF and encoding to sort out, but adapting the
> wording from the XML specification doesn't seem too hard.

Might be nice to have.  What are the implications?

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

---

From: Ovid <[email-masked]>
Date: Tue, 2 Mar 2010 08:43:29 -0800 (PST)

>   http://github.com/gaurav/test-anything-protocol/blob/master/draft-vaidya-test-anything-protocol-00.xml

First off, I think this is great work.  However, there are a few things I would like to point out.

1.  I assume the fact that it doesn't mention nested TAP is considered a feature?  Some of the descriptions are incorrect for nested TAP (e.g., only one plan -- unless it's made clear the plan is left-justified for its scope).

2.  Speaking of left-justified, I admit to not being overly familiar with ABNF and could be wrong, but it doesn't make it clear that the lines must left-justified.  This has significant implications for nested TAP.

3.  Plan TODO:

  Plan-Todo = Plan-Simple "todo" 1*(SP Test-Number) ";"

Wasn't that obsoleted?  No one uses it (that I'm aware of) and it turns out to be very fragile (though, annoyingly, it appears that TAP::Parser still supports this).

4.  There appears to be some inconsistency in skip and todo casing.  Specifically, they're case insensitive.  # TodO is annoying, but acceptable.

5.  Whitespace:

  Test-Result = Status [SP Test-Number] [SP Description]
                  [SP "#" SP Directive [SP Reason]] EOL

According to the ABNF spec (http://tools.ietf.org/html/rfc5234#appendix-B.1), SP is %x20 -- a single space.  Multiple spaces (sometimes no spaces) are allowed in various spots. I would suggest reading TAP::Parser::Grammar code. It indicates this and shows many examples of real-world fixes put in there.

http://cpansearch.perl.org/src/ANDYA/TAP-Parser-0.54/lib/TAP/Parser/Grammar.pm

The following is legal TAP, but doesn't match the spec:

  1..3
  ok 1
  not ok 2 #ToDo asdf
  ok 3 #    skIp

Cheers,
Ovid

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 17:54:34 +0100 (CET)

This is Re: the current version of the draft, 
<http://github.com/gaurav/test-anything-protocol/blob/master/draft-vaidya-test-anything-protocol-00.xml>

Ovid said:
> Sometimes that's something may want in a description and it reads 
> naturally.  I'd like to see the hash mark (octothorpe, whatever), 
> escaped:
>

FWIW, I think escaping is completely reasonable. It's already 
implemented like this in Test::Builder and Test::Harness, and could at 
least be considered "established practice" in the Perl community. (Not 
sure since when though.)

We have four options, I think.

1) Keep disallowing # in the description (breaking with established
    practice, but perhaps gain a small simplification in the draft
    text.)

2) Allow escaped # in the description, using the \# syntax already
    established in the Perl community's most used testing modules.

3) Allow escaped # in the description, using some other syntax,
    e.g. the printf(3)-like "##" way with double-characters ("%%" to
    represent a literal "%".) This breaks with both established
    practice and doesn't seem to give an obvious benefit otherwise.

I'm for 2).

- Salve

---

From: Gaurav Vaidya <[email-masked]>
Date: Wed, 3 Mar 2010 00:55:55 +0800

Hi Salve,

On Mar 2, 2010, at 11:39 PM, Salve J Nilsen wrote:
> Are we considering this one the authoritative version now? (Just to  
> be clear.)
Let's keep http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard 
  as the authoritative version, so it's easier to edit. Once we've  
worked the last few TODOs out and everybody's happy with the wordings,  
we can switch over to tweaking the XML file until it's Perfect (tm).  
Going from the wiki to the XML file takes some work, but it's not  
much: a day, tops, although I guess tweaking will probably go on for a  
while afterwards.

> I think having an established MIME type for TAP streams would be  
> very useful, and definitely worth jumping through some hoops for.
Right. I'll get cracking on this when I've tuits next, which will  
hopefully be this weekend. It's all there in RFC 4288 somewhere :-P.

cheers,
Gaurav

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 17:58:37 +0100 (CET)

> We have four options, I think.

0) Don't care and leave it as it is. Deal with any consequences
    whenever they crop up.

>   practice and doesn't seem to give an obvious benefit otherwise.

0) is not much of an option though. :)

- Salve

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 18:11:30 +0100 (CET)

Ovid said:
> 1.  I assume the fact that it doesn't mention nested TAP is
>     considered a feature?  Some of the descriptions are incorrect
>     for nested TAP (e.g., only one plan -- unless it's made clear
>     the plan is left-justified for its scope).

Could you remind us a little of what the purpose of Nested TAP was?

- Salve

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 18:18:34 +0100 (CET)

Ovid said:
> 2.  Speaking of left-justified, I admit to not being overly familiar
>     with ABNF and could be wrong, but it doesn't make it clear that
>     the lines must left-justified.  This has significant
>     implications for nested TAP.

Would you mind giving us a little context here? Are we considering 
comments written in right-to-left languages like Hebrew or something?

</confused>

- Salve

---

From: Ovid <[email-masked]>
Date: Tue, 2 Mar 2010 09:20:56 -0800 (PST)

> Could you remind us a little of what the purpose of Nested

The main purpose is that it allows multiple TAP streams to be combined into a single stream.  It can be nested to arbitrary depths, but all nesting depths must be consistent.  The first "unindented" line under each nest is the pass/fail status of the nest.  A nest is represented as a single pass/fail result for the outer stream.

  1..3
  ok 1
  ok 2
    ok 1
    not ok 2
    not ok 3
    ok 4
    1..4
  not ok 3 - This summarizes above results

So the outer stream has three tests, one of them being the summary line for the preceding nest.

Also, note that the top-level stream has the plan before, but the nest has the plan after.  Some have found this useful when they want a "subtest" where they don't know how many tests are run, but they still want that leading plan.

Conceptually this maps well to xUnit where a test method is a single test with multiple asserts.  With nested TAP, the nest is equivalent to the asserts and the summary test line is equivalent to the test method.

One unanswered question regarding nested TAP: are skip and todo allowed on summary test lines?

Cheers,
Ovid

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 18:32:51 +0100 (CET)

Gaurav Vaidya said:
> Let's keep 
> http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard 
> worked the last few TODOs out and everybody's happy with the 
> wordings, we can switch over to tweaking the XML file until it's 
> Perfect (tm).

Fair enough. The wiki page is the autoritative version until we're 
happy with it.

- Salve

---

From: Salve J Nilsen <[email-masked]>
Date: Tue, 2 Mar 2010 18:33:56 +0100 (CET)

> hopefully be this weekend. It's all there in RFC 4288 somewhere :-P.

Seems there are a few hoops there, yes. In the meantime, should we
just declare application/x-tap as our "temporary" MIME type? :)

- Salve

---

From: Ovid <[email-masked]>
Date: Tue, 2 Mar 2010 09:42:05 -0800 (PST)

> Would you mind giving us a little context here? Are we
> considering comments written in right-to-left languages like
> Hebrew or something?

I'm not overly familiar with ABNF, so maybe it's clear from the grammar, but since EOL appears to be explicit, I think this is a non-issue.  I just wanted to make it clear that the following isn't legal:

  | 1..1
  | ok 1

(Assumes '|' is the beginning of the line.)

Note the space between that and the plan and test.  The above could be legal if an only if it's a proper nest.  Again, I think this is a non-issue.

Cheers,
Ovid

---

From: Salve J Nilsen <[email-masked]>
Date: Fri, 5 Mar 2010 17:12:25 +0100 (CET)

Ovid said:
> The main purpose is that it allows multiple TAP streams to be 
> combined into a single stream.  It can be nested to arbitrary 
> depths, but all nesting depths must be consistent.  The first 
> "unindented" line under each nest is the pass/fail status of the 
> nest.  A nest is represented as a single pass/fail result for the 
> outer stream.

[snip explanation]

> allowed on summary test lines?

SKIP means "I expect this test to fail, so I won't run it, but it's 
result should be read as ok anyway." If a set of subtests should be 
summarized with "SKIP", we'll have to know a few things about it:

1) Are ALL the subtests "SKIPped"?

  - In this case I think it's reasonable to summarize with a "ok #
    SKIP" if all subtests were ok, or a "not ok # SKIP" if one of them
    failed.

2) Are SOME of the subtests SKIPped, while others are regular tests?

  - A pass or a fail in the regular subtests here would throw off the
    summary. If all SKIPped subtest pass, but one regular subtest
    fails, should this be summarized as "not ok # SKIP" or "not ok"?
    Same question goes if both a SKIPped and a regular subtest fails.

  - If all subtests succeed, should it be reported as "ok" or "ok #
    SKIP"?

  - One way to clean up the summary is to define a "summary priority."
    Which reports are the most important/general?
    (E.g. regular > SKIP > TODO; the most "important" fail ends up in
    the summary)

  - Another way is to require a warning when the subset result is
    ambiguous

  - A third way is to extend the protocol to give a more detailed
    report on what the subtest results were. :-P

3) Are NONE of the subtests SKIPped?

  - Summarizing a subset with no SKIPs as SKIPped sounds very much like
    an error to me. :)

I think the case for TODOs are pretty similar, which leaves us with 
the question "How to summarize a mixed result?"

- If all subtests pass, then the summary should be a pass.
- If at least one regular (non-SKIP, non-TODO) subtest fails, then the
   entire subset is summarized with a fail.
- If at least one SKIP subtest fails, but no regular tests, then the
   summary should be a failed SKIP (not ok # SKIP.) -- Treat it as
   one SKIPped test.
- If at least one TODO subtest fails, but no regular tests or SKIPped
   tests fail, then the subset should be reported as passed, with a
   warning ("not ok # TODO Reason".) -- Treat it as one TODO test.
- If there is a mismatch between subtests and summary (e.g. summary is
   given as a failed TODO, but there are failed SKIP tests) then this
   should be a protocol error.

- Salve, throwing in some thoughts to get things going.. :)

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Mon, 15 Mar 2010 14:14:02 +0100

* Salve J Nilsen <[email-masked]> [2010-03-02 18:25]:
> Gaurav Vaidya said:
> > It's all there in RFC 4288 somewhere :-P.
>
> Seems there are a few hoops there, yes. In the meantime, should
> we just declare application/x-tap as our "temporary" MIME type?
> :)

Works for me.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>