Subject: [tap] Indicating errors in the test suite

From: Salve J Nilsen <[email-masked]>
Date: Fri, 5 Mar 2010 17:37:09 +0100 (CET)

Hey!

Just wanted to get some clarity, to see if I've understood things 
correctly.

In any test suite we have basically two error conditions:

1) Errors in the code that's being tested with the test suite. This is
    the main purpose of a test suite.

2) Errors in the test suite code itself.

Detecting cases of 1) is pretty straightforward as long as you have 
tests for it, but for detecting 2) we just have four "indirect" 
indicators of their presence:

1) "not ok # SKIP" - We intended to skip, but failed somehow
2) "ok # TODO" - A test unexpectedly succeeded
3) Mismatch between plan and number of tests that were done.
4) Shell return codes that are unexpected.

- Do we have any other indicators?
- Should we say something about subprocess return codes in the TAP
   standard?

- Salve

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <[email-masked]>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)

---

From: Ovid <[email-masked]>
Date: Sat, 6 Mar 2010 05:36:29 -0800 (PST)

----- Original Message ----
> From: Salve J Nilsen <[email-masked]>

> - Should we say something about subprocess return codes in the TAP

Returns codes are unreliable. You don't get them if you're running TAP in Javascript.  If some intermediate process is handing the TAP to the consumer and that intermediate process fails but the code you were testing did not, relying on the return code will give you a false negative.

Cheers,
Ovid --
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog - http://use.perl.org/~Ovid/journal/
Twitter - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

---

From: Salve J Nilsen <[email-masked]>
Date: Sat, 6 Mar 2010 15:40:01 +0100 (CET)

Ovid said:
> Returns codes are unreliable. You don't get them if you're running 
> TAP in Javascript.  If some intermediate process is handing the TAP 
> to the consumer and that intermediate process fails but the code you 
> were testing did not, relying on the return code will give you a 
> false negative.

Would it be sensibel to just say "We expect a non-error return code if 
it's available" and leave it at that? (Basically saying nothing about 
implementation details of the specific environment, but leaving it to 
the producers and consumers to give a warning if an unexpected return 
code shows up somewhere.)

My point is that even if return codes are unreliable they can still be 
useful indicators of something funky going on somewhere, and because 
of this, the TAP document should at least mention it.

E.g. "TAP producers and consumers should emit a warning if any 
non-successful process return code is detected."

Or to say it in a different way, "don't throw away return codes that 
might mean something, even if they're unreliable."

- Salve

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Mon, 15 Mar 2010 13:26:29 +0100

* Salve J Nilsen <[email-masked]> [2010-03-06 15:35]:
> Or to say it in a different way, "don't throw away return codes
> that might mean something, even if they're unreliable."

I agree.

We already have a similar issue in other cases, eg. `ok # TODO`:
sometimes you want to treat those as PASS and sometimes as FAIL,
and the test itself cannot know which interpretation is right,
and the harness, only at that point in time. If you’re archiving
TAP documents, you might use a different interpretation of the
same one test result at different points in time depending on
what you are mining your data for.

So TAP should not come with value judgement included. It should
merely capture outcomes. It is up to the consumer at the point in
time where it is parsing the TAP to make these judgements.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>