Subject: [tap] Weird thought for the day: "skip 71 - reason"

From: Ovid <[email-masked]>
Date: Fri, 5 Mar 2010 07:53:33 -0800 (PST)

One thing which bugs me about TAP is this:

  1..5
  ok 1 - some test
  ok 2 - another test
  ok 3 # skip cuz I said so
  not ok 4 # skip, skip, baby
  ok 5 -final test
 
What does a skipped "not ok" test mean? Its behaviour is undefined. This means that different parsers might implement this in different ways.  It's an obscure edge case, but with greater adoption of TAP, it might crop up.

Here's what I would like to see.

  1..5
  ok 1 - some test
  ok 2 - another test
  skip 3 - skip cuz I said so
  skip 4 - skip, skip, baby
  ok 5 -final test

In other words, you can't introduce ambiguity here.

Though this list is for ietf, those who really care about TAP are hopefully reading this.

Thoughts?  Something corresponding for TODO (less needed due to less ambiguity)?

Cheers,
Ovid--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

---

From: Salve J Nilsen <[email-masked]>
Date: Fri, 5 Mar 2010 16:56:40 +0100 (CET)

Ovid said:
>
> What does a skipped "not ok" test mean? Its behaviour is undefined. 
> This means that different parsers might implement this in different 
> ways.  It's an obscure edge case, but with greater adoption of TAP, 
> it might crop up.

http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#The_.22SKIP.22_directive

   "Since the test was skipped, the test result is expected to be "ok"
   (indicating that the test was skipped correctly). A skipped test
   with a result of "not ok" is suspicious, and the TAP consumer should
   report a warning."

I think this is pretty clear.... It means "something funky is going on 
in your test suite."

- Salve

-- 
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print#  Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.#     <[email-masked]>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}";   __END__ is near! :)

---

From: Andy Armstrong <[email-masked]>
Date: Fri, 5 Mar 2010 16:03:45 +0000

> Thoughts?  Something corresponding for TODO (less needed due to less ambiguity)?

It makes more sense - but raises a wider question about how much of the grammar we're prepared to change. Are we saying that for TAP v14 we can basically change anything?

-- 
Andy Armstrong, Hexten

---

From: Ovid <[email-masked]>
Date: Fri, 5 Mar 2010 08:12:42 -0800 (PST)

----- Original Message ----
> From: Andy Armstrong <[email-masked]>

> It makes more sense - but raises a wider question about how much of the grammar 
> we're prepared to change. Are we saying that for TAP v14 we can basically change 
> anything?

I think a TAP consumer could conceivably indicate to the producer which version of TAP it supports.  The produce is not bound by this and should produce TAP compatible with version 13. If not version is specified, again, 13 (if 14 is not backwards compatible).  Otherwise, if the producer and consumer can produce/understand richer TAP levels, we have a win.

That being said, I don't think "skip 3" is a compelling enough argument for this.

And Salve, I just wasn't aware this. I remember old conversations where it was agreed that the behaviour is unspecified.
 
I would just like to get people thinking about what we can do, rather than keep our thinking small (though my modest proposal was rather small :)

Cheers,
Ovid--

---

From: Steffen Schwigon <[email-masked]>
Date: Sun, 07 Mar 2010 12:40:29 +0100

Ovid <[email-masked]> writes:
> What does a skipped "not ok" test mean? Its behaviour is
> undefined. This means that different parsers might implement this in
> different ways.  It's an obscure edge case, but with greater
> adoption of TAP, it might crop up.

I do not have a problem with this semtantic gap, I even already used
it, similar to what #TODO does.

That case for instance happens when Test::Builder toolchains are not
available and it is easier to canonically independently generate the
OK/NOTOK part and the directives.

> In other words, you can't introduce ambiguity here.

“NOT OK # SKIP” is no ambiguity. It does not conflict with anything
else. It is skipped and semantically being “OK”, same as TODO.

The interpretation as SKIP by the TAP::Parser/Aggregator should just
be handled the same emotionless, statistic way as TODO: just count how
many SKIP, and if you like then also how many of them are unexpectedly
OK/NOTOK and left the “semantic strangeness” as exercise to the user.

I *strongly* vote *against* introducing a new value additionally to
OK/NOT_OK. Especially as it doesn't take away the ambiguity of the
“NOT OK # SKIP” case and would be an artificial change introduced
without real urge from users (others than the “Wizards Of TAP”).

I regularly teach TAP to users and the whole point of it's easyness is
the fact I can tell:

  Basically you either report OK or NOT OK (and a plan). 
  That's all. The rest is just sugar.

And I would like to keep the basics and the sugar divided as it
currently is: basics on the left side, and the more sugar the more on
the right.

TAP is KISS: Keep It Simple, Stupid; with emphasis on “keep”.

For non-Perl, non-Test::Builder toolchains that easiness is vital!

> Though this list is for ietf, those who really care about TAP are

I do care for TAP. 
I use it. 
I teach it.
I build infrastructures around it. 
Perl and non-Perl. In harmony. Quite.

I hope that qualifies my comments a bit. :-)

Kind regards,
Steffen 
-- 
Steffen Schwigon <[email-masked]>

---

From: Steffen Schwigon <[email-masked]>
Date: Sun, 07 Mar 2010 12:42:12 +0100

> I think this is pretty clear.... It means "something funky is going on

I agree. That's ok and enough.
I put more explanations in my neighbor post.

Kind regards,
Steffen

---

From: Salve J Nilsen <[email-masked]>
Date: Wed, 10 Mar 2010 20:55:28 +0100 (CET)

Ovid said:
>
> I would just like to get people thinking about what we can do, 
> rather than keep our thinking small (though my modest proposal was 
> rather small :)

Ok, I'll bite.

What I'd love to see, is any necessary support in the protocol to let 
me save the complete results of a test run together with any related 
information that might be necessary to figure out why a failure 
happened.

I vaguely remember we've talked about this at some occation (a JSON or 
YAML based TAP metadata spec). Did anything happen there?

- Salve

---

From: Michael Peters <[email-masked]>
Date: Wed, 10 Mar 2010 15:01:44 -0500

On 03/10/2010 02:55 PM, Salve J Nilsen wrote:

> What I'd love to see, is any necessary support in the protocol to let me
> save the complete results of a test run together with any related
> information that might be necessary to figure out why a failure happened.

This is why I created TAP::Harness::Archive. A TAP archive is basically 
a tarball of tap streams and a META.yml file which describes the test 
run. This includes in what order the files ran in, how long they took to 
run, etc. It also allows any extra files that might be important to 
tracking down a failure (like log files, etc).

-- 
Michael Peters
Plus Three, LP

---

From: Salve J Nilsen <[email-masked]>
Date: Wed, 10 Mar 2010 21:08:22 +0100 (CET)

Michael Peters said:
>> What I'd love to see, is any necessary support in the protocol to 
>> let me save the complete results of a test run together with any 
>> related information that might be necessary to figure out why a 
>> failure happened.
>
> This is why I created TAP::Harness::Archive. A TAP archive is 
> basically a tarball of tap streams and a META.yml file which 
> describes the test run. This includes in what order the files ran 
> in, how long they took to run, etc. It also allows any extra files 
> that might be important to tracking down a failure (like log files, 
> etc).

Right. Are there any specific features in the TAP stream you're 
missing? (E.g. system information, uname(1) output, timestamps, perl 
version, module versions, ..)

Or approaching from another directions: Do you know of any very useful 
features available some continuous integration frameworks (or similar) 
that could me made better or possible with the right kind of 
environment/meta information from each single test run?

- Salve

---

From: Michael Peters <[email-masked]>
Date: Wed, 10 Mar 2010 15:17:24 -0500

On 03/10/2010 03:08 PM, Salve J Nilsen wrote:

> Right. Are there any specific features in the TAP stream you're missing?
> (E.g. system information, uname(1) output, timestamps, perl version,
> module versions, ..)

There are some things that could be done to remove some of what a TAP 
archive does, but not all of it, or at least without turning TAP into 
some kind of monster.

+ Nested TAP could let us aggregate multiple TAP streams into a single 
TAP stream to represent a single test suite run. But then you aren't 
dealing with the TAP as the test spit it out. Instead as some 
preprocessor assembled it.

+ TAP meta data could be added to a tap stream to record things like 
uname, timing, etc. But unless this is done on a nested TAP stream that 
represents a full test run, then this information only captures an 
individual stream's experience.

+ Extra files (eg, Apache log files): Where would you put them? Trying 
to somehow include them in the TAP stream seems like a really bad idea. 
You'd need some sort of encode/decode so they don't clash with other 
stuff and then they become completely unreadable.

> environment/meta information from each single test run?

If you create a TAP archive with extra files then you can already use 
this in Smolder. Upload it and then you can browse those extra files 
when looking at failed tests. We use it at $work a lot.

-- 
Michael Peters
Plus Three, LP

---

From: Salve J Nilsen <[email-masked]>
Date: Wed, 10 Mar 2010 22:58:32 +0100 (CET)

Michael Peters said:
>> missing? (E.g. system information, uname(1) output, timestamps, 
>> perl version, module versions, ..)
>
> There are some things that could be done to remove some of what a 
> TAP archive does, but not all of it, or at least without turning TAP 
> into some kind of monster.
>
> + Nested TAP could let us aggregate multiple TAP streams into a 
> single TAP stream to represent a single test suite run. But then you 
> aren't dealing with the TAP as the test spit it out. Instead as some 
> uname, timing, etc. But unless this is done on a nested TAP stream 
> that represents a full test run, then this information only captures 
> an individual stream's experience.

Couldn't the same tap stream "linking" be achieved just by assigning a 
"test run id" that is consistent across streams in the same test run?

>> Or approaching from another directions: Do you know of any very 
>> useful features available some continuous integration frameworks 
>> (or similar) that could me made better or possible with the right 
>> kind of environment/meta information from each single test run?
>
> If you create a TAP archive with extra files then you can already 
> use this in Smolder. Upload it and then you can browse those extra 
> files when looking at failed tests. We use it at $work a lot.

I'm asking more about what _can't_ be done with Smolder, and if some 
new aggregatable information could make it easier to implement these 
features at some point. E.g. test run timing, system load at end of 
run, I/O usage, etc. Once you have a few months of collected data you 
can figure out new and non-obvious things.

- Salve

---

From: Aristotle Pagaltzis <[email-masked]>
Date: Mon, 15 Mar 2010 14:02:15 +0100

* Ovid <[email-masked]> [2010-03-05 17:05]:
> I think a TAP consumer could conceivably indicate to the
> producer which version of TAP it supports. The produce is not
> bound by this and should produce TAP compatible with version
> 13. If not version is specified, again, 13 (if 14 is not
> backwards compatible). Otherwise, if the producer and consumer
> can produce/understand richer TAP levels, we have a win.

I really dislike the idea of introducing any sort of handshake.

TAP works in environments where emitter and consumer are removed
far enough from each other that no handshake is possible, and it
should continue to be able to work in those. It also works for
environments where it is too difficult to implement any sort of
protocol in the emitter (eg. on a microcontroller).

TAP should let the emitter be as near braindead as possible. The
consumer is the place to burden with any resulting complexity.
The consumer can be assumed to run in such an environment lavish
with CPU and memory resources that the emitter cannot.

This is connected with the “desciptive not prescriptive” point
too, in that TAP should leave the consumer’s job to the consumer,
providing pure data capture, as comprehensive as the emitter can
manage and afford.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

---

From: Andy Armstrong <[email-masked]>
Date: Mon, 15 Mar 2010 13:36:37 +0000

> I really dislike the idea of introducing any sort of handshake.

I agree.

-- 
Andy Armstrong, Hexten

---

From: Ovid <[email-masked]>
Date: Mon, 15 Mar 2010 07:14:43 -0700 (PDT)

> > I really dislike the idea of introducing any sort of

I'm convinced.

Cheers,
Ovid