Subject: [tap] interpretation of results

From: Michael Peters <[email-masked]>
Date: Wed, 04 Mar 2009 10:30:50 -0500

Given this TAP:

1..3
ok 1
not ok 2

Does it mean that 1 of 2 tests failed and we had a bad plan? Or does it mean the 
plan was right, but only 1 of 3 tests passed. Or in other words, is there a 3rd 
implicit failed test in there?

Smolder currently shows the user that 1 or 2 tests failed. Just wondering if 
this should be changed.
http://sourceforge.net/tracker/?func=detail&atid=818650&aid=2658074&group_id=161136

-- 
Michael Peters
Plus Three, LP

---

From: Andy Armstrong <[email-masked]>
Date: Wed, 4 Mar 2009 15:46:17 +0000

On 4 Mar 2009, at 15:30, Michael Peters wrote:

> Given this TAP:
>
> 1..3
> ok 1
> not ok 2
>
> Does it mean that 1 of 2 tests failed and we had a bad plan? Or does  
> it mean the plan was right, but only 1 of 3 tests passed. Or in  
> other words, is there a 3rd implicit failed test in there?

We planned three, ran two and one of them failed. I think that  
treating the missing test as a fail would be something you'd only do  
if you wanted to distill the results down to a simple failure count.

-- 
Andy Armstrong, Hexten

---

From: Michael G Schwern <[email-masked]>
Date: Wed, 04 Mar 2009 13:15:40 -0800

Andy Armstrong wrote:
>> it mean the plan was right, but only 1 of 3 tests passed. Or in other
>> words, is there a 3rd implicit failed test in there?
> 
> We planned three, ran two and one of them failed. I think that treating
> the missing test as a fail would be something you'd only do if you
> wanted to distill the results down to a simple failure count.

Agreed.  It's important to distinguish between a test failing and a test never
having run at all.

-- 
Insulting our readers is part of our business model.
        http://somethingpositive.net/sp07122005.shtml

---

From: Ovid <[email-masked]>
Date: Wed, 4 Mar 2009 23:50:53 -0800 (PST)

----- Original Message ----

> From: Michael G Schwern <[email-masked]>

> Agreed.  It's important to distinguish between a test failing and a test never

Also agreed.  As a more general programming rule, it's rarely a good idea to infer meaning from the absence of information.  Just look at how a NULL salary value could be abused in a database.  Are they actually hourly and not salary?  Is there a job offer and they haven't agreed to terms yet?  Are they an unpaid intern?  Inferring meaning from the absence of information means that cases like this get forgotten and technical debt accrues while people build in hacks to get around stuff like that.

Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6