Subject: Re: [tap] RFC Status?

From: Leon Timmermans <[email-masked]>
Date: Wed, 17 Sep 2014 16:24:33 +0200

On Fri, Aug 15, 2014 at 3:07 AM, Gaurav Vaidya <[email-masked]> wrote:

> Hullo!
>
> On 13 Aug, 2014, at 12:09 pm, Andrew de Andrade <[email-masked]>
> wrote:
>
> > On Tue, Aug 12, 2014 at 1:47 PM, Jonathan Kingston <
> [email-masked]> wrote:
> >> I have created a new repo(https://github.com/TestAnything/Specification)
> for TAP which can be used to track changes better, conversations and also
> building TAP 14 separate to the site itself which I see as a published
> authority.
> >
> > Instead of creating a brand new repo with no history, just clone
> --mirror guarav's repo and --mirror push it to TestAnything/Specification.
> His version already has a started RFC and has commits he's already made
> that should be preserved. Alternatively, maybe Guarav can just transfer
> ownership to the TestAnything organization.
> > https://github.com/gaurav/test-anything-protocol/
> >
> > Guarav, are you ok with transferring ownership to the TestAnything
> github org? This would have the spec in one central location where everyone
> can fork from.
>
> Absolutely! Apparently, I need to be an admin on TestAnything to do the
> transfer, so could someone add me there (username: gaurav)? I’ll give up
> the admin bit as soon as I get
> https://github.com/gaurav/test-anything-protocol/ transferred.
>

*Crickets*

Leon

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 08:59:39 -0700

Yeah, who's responsible for making this happen? I would love to see it.
 
Best,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas - http://www.overseas-exile.com/

On Wednesday, 17 September 2014, 16:25, Leon Timmermans <[email-masked]> wrote:

>>On 13 Aug, 2014, at 12:09 pm, Andrew de Andrade <[email-masked]> wrote:
>>
>>> On Tue, Aug 12, 2014 at 1:47 PM, Jonathan Kingston <[email-masked]> wrote:
>>>> I have created a new repo(https://github.com/TestAnything/Specification) for TAP which can be used to track changes better, conversations and also building TAP 14 separate to the site itself which I see as a published authority.
>>>
>>> Instead of creating a brand new repo with no history, just clone --mirror guarav's repo and --mirror push it to TestAnything/Specification. His version already has a started RFC and has commits he's already made that should be preserved. Alternatively, maybe Guarav can just transfer ownership to the TestAnything organization.
>>> Guarav, are you ok with transferring ownership to the TestAnything github org? This would have the spec in one central location where everyone can fork from.
>>
>>Absolutely! Apparently, I need to be an admin on TestAnything to do the transfer, so could someone add me there (username: gaurav)? I’ll give up the admin bit as soon as I get https://github.com/gaurav/test-anything-protocol/ transferred.
>>
>
>
>*Crickets*
>
>Leon
>
>
>_______________________________________________
>tap mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/tap
>
>
>

---

From: Leon Timmermans <[email-masked]>
Date: Wed, 17 Sep 2014 20:07:49 +0200

On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <[email-masked]>
wrote:

> I'm working on some testing libraries for JavaScript based on TAP.
> Specifically, I'm trying to explore making it possible for library authors
> to not only check the results of their TAP tests, but also map-reduce the
> TAP results of the dependents of their library to see how changes to their
> library impact the correctness of their dependents.
>
> e.g. I am the author of library A. I publish version 0.1.0. Another author
> includes my library as a dependency in library B and in library C and uses
> TAP to test those libraries. Given this scenario, it would be nice if I
> could, when modifying my library, run not only my tests, but also run the
> tests of all my dependents built with my previous version and my current
> version and compare how my changes impacted the "correctness" of their
> programs.
>
> Beyond the benefits to a community, helping authors know when they can
> likely safely upgrade their deps, this type of runner would also help in an
> corporate environment with lots of code re-use between teams.
>
> While thinking about this, I decided to go out and figure out if there is
> such a thing as nested TAP in the protocol specification. While searching
> for this, I came across this group, which is a great, but it appears there
> has been no activity here in a while and it looks like all the wiki pages
> with prior information about the state of the TAP protocol becoming an IETF
> RFC have disappeared.
>
> What's the current state of TAP? Is there still interest in taking this to
> RFC status? I found some previous discussions on the list about nested TAP.
> Did those make it into the specification in any way? If so, where can I
> find examples of correct nested tap results?
>
> Furthermore, is there a standardized set of test fixtures for the current
> version that any implementation can be tested against for correctness and
> performance? Having worked a bit with JSON schema, I found that one of the
> most useful tools was a standardized set of tests to check if a particular
> implementation conforms to the protocol standard. (see:
> https://github.com/json-schema/JSON-Schema-Test-Suite )
>
>
I've been recently writing a TAP Harness, and for subtests I followed what
Perl has been doing for years (just like TAP::Stream): have indented
subtests follow by a non-indented summary line. None of this is described
in detail anywhere though :-/, nor is the interaction with the barely
defined YAMLish well-explored AFAIK.

Leon

---

From: Andrew de Andrade <[email-masked]>
Date: Wed, 17 Sep 2014 11:35:30 -0700

Sorry for being MIA (burning man, visiting family). I was planning on
getting on this after starting my new job (started on monday). I've got ~2
weeks of onboarding tasks to handle, then I will work on what I said I
would above. It looks like I might be responsible for front-end testing
here, so that should help :)

On Wed, Sep 17, 2014 at 11:07 AM, Leon Timmermans <[email-masked]> wrote:

> On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <
> [email-masked]> wrote:
>
>> e.g. I am the author of library A. I publish version 0.1.0. Another
>> author includes my library as a dependency in library B and in library C
>> and uses TAP to test those libraries. Given this scenario, it would be nice
>> if I could, when modifying my library, run not only my tests, but also run
>> the tests of all my dependents built with my previous version and my
>> current version and compare how my changes impacted the "correctness" of
>> their programs.
>>
>> What's the current state of TAP? Is there still interest in taking this
>> to RFC status? I found some previous discussions on the list about nested
>> TAP. Did those make it into the specification in any way? If so, where can
>> I find examples of correct nested tap results?
>>

-- 
Q: Why is this email four sentences or less?
A: http://four.sentenc.es

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 12:08:49 -0700

A few notes on this.

For subtests, they are indented four spaces. Always four spaces. If they are indented eight spaces, you know they're a subtest of a subtest (assuming there's an intervening subtest. That means this is valid TAP, but there is no subtest because it's indented too far:
ok 1 - foo
>        ok 1 - bar indented too far
>ok 2 - bar passed
The "ok 1 - bar" line would be discarded as "unknown". 

So if, in a stream of TAP, you find something which would be valid tap if it were unindented by four spaces, it's a subtest. The trailing, unindented summary line must mirror the status of the subtest. Thus, the following is invalid:

ok 1 - foo
>    not ok 1 - bar indented too far
>ok 2 - bar passed
>
>Currently, Test::Harness treats the above as passing when it should not. I believe Schwern filed a bug for that. I know Andy Armstrong and I both took a quick swing at extending Test::Harness but discovered that it was more difficult than it appeared.

The grammar would be modified to look sort of like this (modifying http://search.cpan.org/dist/Test-Harness/lib/TAP/Parser/Grammar.pm):

line           ::= (comment | test | subtest | unknown | bailout ) "\n"
>subtest        ::= subtest_line { subtest_line } test
>subtest_line   ::= '    ' line
That assumes we're OK with a recursive grammar. Also, the grammar doesn't really have a way of identifying the semantics of the trailing line on the subtest grammar line.

If you have other questions about subtests, I think I can give definitive answers.

As for the YAMLish structured diagnostics, those were never official, as far as I recall. There were several disputes, one of which was whether or not we should use YAML or JSON (I recall even XML being mentioned). Some argued that any structured diagnostic should be allowed, but I reject that firmly because that makes life hell for the parser. If you asked me what my stance is today, I would think that structured diagnostics should valid JSON, but that raises questions about whitespace issues, UTF-8, and so on. If we could agree on the format, then we'd need to agree on the keys in the structured diagnostic dictionary (and how to extend them).

Best,
Ovid

On Wednesday, 17 September 2014, 20:07, Leon Timmermans <[email-masked]> wrote:

>
>
>On Mon, Aug 11, 2014 at 8:42 PM, Andrew de Andrade <[email-masked]> wrote:
>
>I'm working on some testing libraries for JavaScript based on TAP. Specifically, I'm trying to explore making it possible for library authors to not only check the results of their TAP tests, but also map-reduce the TAP results of the dependents of their library to see how changes to their library impact the correctness of their dependents. 
>>
>>
>>e.g. I am the author of library A. I publish version 0.1.0. Another author includes my library as a dependency in library B and in library C and uses TAP to test those libraries. Given this scenario, it would be nice if I could, when modifying my library, run not only my tests, but also run the tests of all my dependents built with my previous version and my current version and compare how my changes impacted the "correctness" of their programs. 
>>
>>
>>Beyond the benefits to a community, helping authors know when they can likely safely upgrade their deps, this type of runner would also help in an corporate environment with lots of code re-use between teams. 
>>
>>
>>While thinking about this, I decided to go out and figure out if there is such a thing as nested TAP in the protocol specification. While searching for this, I came across this group, which is a great, but it appears there has been no activity here in a while and it looks like all the wiki pages with prior information about the state of the TAP protocol becoming an IETF RFC have disappeared. 
>>
>>
>>What's the current state of TAP? Is there still interest in taking this to RFC status? I found some previous discussions on the list about nested TAP. Did those make it into the specification in any way? If so, where can I find examples of correct nested tap results?
>>
>>
>>Furthermore, is there a standardized set of test fixtures for the current version that any implementation can be tested against for correctness and performance? Having worked a bit with JSON schema, I found that one of the most useful tools was a standardized set of tests to check if a particular implementation conforms to the protocol standard. (see: https://github.com/json-schema/JSON-Schema-Test-Suite )
>>
>>
>I've been recently writing a TAP Harness, and for subtests I followed what Perl has been doing for years (just like TAP::Stream): have indented subtests follow by a non-indented summary line. None of this is described in detail anywhere though :-/, nor is the interaction with the barely defined YAMLish well-explored AFAIK.
>
>Leon
>
>

---

From: "Bruno P. Kinoshita" <[email-masked]>
Date: Wed, 17 Sep 2014 12:16:56 -0700

Ovid, 

Would a subtest as follows be valid?

    ok 1 - subtest 1a
    ok 2 - subtest 1b
    1..2
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b
    1..2
ok 2 - Subtest 2
1..2
ok

A user posted this TAP stream generated with Test::More to tap4j [1]. I'll probably end up writing a custom parser for it to support streams generated by Perl tools, but I'd like to know whether it would be considered valid or not.

TIA
Bruno

[1] https://github.com/tupilabs/tap4j/issues/15

>________________________________
> From: Ovid <[email-masked]>
>To: Leon Timmermans <[email-masked]>; Andrew de Andrade <[email-masked]> 
>Cc: "[email-masked]" <[email-masked]> 
>Sent: Wednesday, September 17, 2014 4:08 PM
>Subject: Re: [tap] RFC Status?
> 
>
>
>A few notes on this.
>
>

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 12:40:31 -0700

Bruno,

Yes, those are perfectly valid subtests. I, like an idiot, forgot to show the subtest plans in my examples. 
 
Best,
Ovid
--
IT consulting, training, international recruiting
       http://www.allaroundtheworld.fr/.
Buy my book! - http://bit.ly/beginning_perl
Live and work overseas -
 http://www.overseas-exile.com/

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 12:52:58 -0700

Naturally, I forgot the plans in my example. May as well make it the full grammar now (with lines referring to subtests being bolded and indented, with apologies to those with text-only email clients).

tap            ::= version? { comment | unknown } leading_plan lines
                   |                                           lines trailing_plan {comment}
version        ::= 'TAP version ' positiveInteger {positiveInteger} "\n"

leading_plan   ::= plan skip_directive? "\n"

trailing_plan  ::= plan "\n"

plan           ::= '1..' nonNegativeInteger

line           ::= (comment | test | subtest | unknown | bailout ) "\n"
subtest        ::= ( '    ' version )? 
                   ( '    ' plan subtest_line { subtest_line }
                               | subtest_line { subtest_line } '    ' plan ) test
subtest_line   ::= '    ' line
test           ::= status positiveInteger? description? directive?
status         ::= 'not '? 'ok '

description    ::= (character - (digit | '#')) {character - '#'}

directive      ::= todo_directive | skip_directive

todo_directive ::= hash_mark 'TODO' ' ' {character}

skip_directive ::= hash_mark 'SKIP' ' ' {character}

comment        ::= hash_mark {character}

hash_mark      ::= '#' {' '}

bailout        ::= 'Bail out!' {character}

unknown        ::= { (character - "\n") }

(* POSIX character classes and other terminals *)

digit              ::= [:digit:]

character          ::= ([:print:] - "\n")
positiveInteger    ::= ( digit - '0' ) {digit}
nonNegativeInteger ::= digit {digit}
 
And the above says nothing about structured diagnostics. As I recall, we were going to punt on that until the next version. Any line not parseable by the above was to be considered an "unknown" line to allow for expanding the grammar with future versions. That may or may not have been a good idea, but in practice, I've never been bitten by it in many years of testing. This allows naughty code to spit out crap to STDOUT which gets ignored by the parser. By default, STDERR is always ignored, but diagnostics (comments, in the grammar) were always send to STDERR, meaning that even if we could parse them, the parser would never see them. That was problematic.

Best,
Ovid

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 12:54:47 -0700

(Am I the last person on the planet who uses Yahoo! Mail? I use it because it's so much of a pain to change the damned thing, but it, again, corrupted the formatting on my mail message. Resending the grammar in hopes that it comes through OK this time)

hash_mark      ::= '#' {' '}

digit              ::= [:digit:]

Best,
Ovid

---

From: "Bruno P. Kinoshita" <[email-masked]>
Date: Wed, 17 Sep 2014 13:12:13 -0700

Thanks Ovid!

Probably when we start discussing subtests I'll chip in with many questions. As, am I allowed to have subtests before and after test results?

1..2
ok 1 - foo
    not ok 1 - some text

ok 2 - bar passed
    not ok 1 - another text

1..2
    not ok 1 - some text

ok 1 - foo
    not ok 1 - another text

ok 2 - bar passed
    not ok 1 - another text

Because my first guess was that subtests would always come after test elements, and be linked to the previous element. 

Thanks again
Bruno

> From: Ovid <[email-masked]>
>To: Bruno P. Kinoshita <[email-masked]> 
>Sent: Wednesday, September 17, 2014 4:40 PM
>Subject: Re: Valids subtests (was: [tap] RFC Status?)
> 
>
>
>Bruno,
>
>

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 13:18:03 -0700

See my grammar in another thread. A subtest is indented for spaces, either begins or ends with a plan (and could start with an optional version number) and the test line immediately following the subtest must have an ok/not ok status to summarize the pass/fail status of the subtest. That ensures that even if older parsers can't recognize the subtest, they'll correctly pick up the summary line and still produce valid results. 
 
Best,
Ovid

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 13:33:00 -0700

Bruno,

I realized that referring you to the grammar is the same as RTFM and that's not fair when there are questions about the FM :)

    1..2
    ok 1 - foo
        not ok 1 - some text
    ok 2 - bar passed
        not ok 1 - another text

The above is not valid because there must always be summary ok/not ok lines *after* the subtest and the subtest must have a leading XOR trailing plan (in other words, not both) with the same indentation level as the subtest and the summary line *must* have an ok/not ok status that matches the results of the subtest. The following would be valid:

    1..2
        not ok 1 - some text
        1..1
    not ok 1 - foo (because the subtest did not pass)
        1..2
        ok 1 - another text
        not ok 2 - whee! # TODO (todo tests are always considered as passing)
    ok 2 - bar passed (because the subtest did pass)

Best,
Ovid

---

From: "Bruno P. Kinoshita" <[email-masked]>
Date: Wed, 17 Sep 2014 13:38:09 -0700

Haha, thanks Ovid!

Well, turns out tap4j TAP parser is likely broken for subtests then :)

I'll probably spend some time writing a new parser - which I'll try to make more compatible with Perl tools and with existing grammars. 

But some definitive documentation for subtests in TAP website would be grand too.

Cheers
Bruno

>Sent: Wednesday, September 17, 2014 5:33 PM
>Subject: Re: [tap] Valids subtests (was:  RFC Status?)
> 
>
>
>Bruno,
>
>

---

From: Ovid <[email-masked]>
Date: Wed, 17 Sep 2014 13:40:22 -0700

Bruno,

Is there a git project for this anywhere? I haven't written any Java for a while and with my company, I don't have a lot of spare time, but I'd be happy to take a look.
 
Best,
Ovid

---

From: "Bruno P. Kinoshita" <[email-masked]>
Date: Wed, 17 Sep 2014 14:02:21 -0700

The source code is hosted at [1]. The web site is generated during the release process at [2]. The test app is deployed at [3] (though it is outdated and needs to be updated). 

I first tried writing a parser by taking a look at snake-yaml, but ended up depending a lot on regex's. So probably that's a part that could be reworked to better represent grammars such as the one you posted earlier. 

Please, feel free to submit issues, pull requests, comments/suggestions :)
Thanks!

[1] https://github.com/tupilabs/tap4j
[2] http://tap4j.org

[3] http://instanttap.appspot.com/

>Sent: Wednesday, September 17, 2014 5:40 PM

---

From: Andrew Rodland <[email-masked]>
Date: Thu, 18 Sep 2014 23:58:52 -0400

> A few notes on this.
>
> For subtests, they are indented four spaces. Always four spaces. If they are
> indented eight spaces, you know they're a subtest of a subtest (assuming
> there's an intervening subtest. That means this is valid TAP, but there is
> no subtest because it's indented too far:
>
> ok 1 - foo

I'm not trying to argue against established behavior, but I'd like to
point out that this disagrees with what you expressed in the previous
thread "More uncertainty about junk lines". In that, we agreed that it
was desirable that the determination of whether a line was
junk/unknown wasn't context sensitive. If it's potentially valid TAP
in any context, it's interpreted as that, and if that causes the TAP
document to be invalid, then so be it. If that holds true, then this
snippet is an error; it has a result in a subsubtest that lacks a plan
or a containing subtest.

Andrew

---

From: Ovid <[email-masked]>
Date: Fri, 19 Sep 2014 01:16:59 -0700

Hi Andrew,

In that, we agreed that it was desirable that the determination of whether a line was junk/unknown wasn't context sensitive.

I've tried searching for where I said that, but I can't find it. Can you provide me with the quote so I can understand where the confusion lies? I may very well be mistaken here.

The reasoning works like this: historically it has always been the case that ok/not ok status is bound to the beginning of the string and even a single space in front of ok/not ok invalid. With a subtest, we make an exception if an only if it is indented exactly four spaces (in the context of the containing scope) and is parseable as TAP. Otherwise, writing a parser could be a nightmare. Consider:

> not ok
> 1..3

Most people aren't aware of this, but that's a perfectly valid (but failing) TAP stream. While it's not recommended, the test numbers have always been optional. Here's the relevant line from the TAP grammar:

Because TAP is designed to be both human-readable and computer-parseable, there exists the unfortunate problem that sometimes it's just a touch harder to parse than normal and some of the plans for it might make life more difficult. For example, if we followed your suggestion, we would then have this scenario:

>        ok this is working now
> ok bar passed

As currently described (by my understanding), the above is valid because the "ok this is working now" line is indented too far and thus discarded. This makes parsing much simpler. However, you argue that the above is invalid because the "ok" line is valid TAP in an invalid context. That makes sense, but consider the plans for structured diagnostics:

> not ok # TODO darn it!
>     --- 

>     file     : t/foo.t
>     datetime : 2015-01-23 01:23:45
>     have     : 7

>     want     : 8
>     comment  : |

>                 We still have an off by one error and it's not
>                 ok that we still haven't tracked it down
>     ...
> ok bar passed

Parsers are not required to parse structured diagnostics (this allows older parsers to ignore them), but that last line of the comment above could be considered valid TAP in an invalid context and result in a parse error, even though the above is perfectly fine.

The one thing I really don't want to see is for TAP to go down the YAML road of having a mind-bogglingly huge spec to cover all sorts of corner cases. This will block TAP adoption and make it much harder for everyone to get right. However, if we are strict about the four space rule and discarding unknown lines, any parser not compliant with that rule can (sort of) be ignored, but that means better communication about requirements.

In a test hackathon in Norway, we discussed creating a TAP document that would be a series of TAP snippets and how they should be parsed. The document would be YAML or something similar and should allow TAP implementors in various languages to parse the TAP snippets and compare their results to the document results. In other words, a language-agnostic test specification. Maybe now would be a good time to create one?

Best,
Ovid

---

From: "Bruno P. Kinoshita" <[email-masked]>
Date: Fri, 19 Sep 2014 05:46:19 -0700

Hello Ovid, 

> In a test hackathon in Norway, we discussed creating a TAP document that would be a series of TAP snippets and how they should be parsed. The document would be YAML or something similar and should…

+1 for that, it would incrase compatibility between producer/consumers in different programming languages.

And we could derive from it a simple web page like JSON-LD has [1] and provide a more readable version as well.

Cheers, 
Bruno

[1] http://json-ld.org/test-suite/

>To: Andrew Rodland <[email-masked]> 
>Sent: Friday, September 19, 2014 5:16 AM

---

From: Steffen Schwigon <[email-masked]>
Date: Fri, 19 Sep 2014 17:59:41 +0200

Ovid <[email-masked]> writes:
> For subtests, they are indented four spaces. Always four spaces.

Bummer! This was an eye-opener for me, in a sense that IMHO with this
nested TAP we step into a nested hell of context sensitive whitespace
parsing, which, as additional bonus, isn't human-readable anymore, and,
as additional additional bonus, already evidently seems to be hard to
implement.

I might exaggerate here slightly, but seriously, is it really worth?

Which use-case can't live with today's TAP when structuring its tests in
separate test files, respective TAP streams.

> As for the YAMLish structured diagnostics, those were never official,
> [...]
> whether or not we should use YAML or JSON
> [...]
> If you asked me what my stance is today, I would think that
> structured diagnostics should valid JSON

And here I seriously start to panic if that opinion should ever lead to
deprecating YAMLish, as 1) it much better fits the philosophy of TAP's
line-based'ness, 2) is easier to generate with line based tools, like in
a shell-only world, and 3) zillions of test results are already based on
it for the last 6 years. I'm not exaggerating here.

Kind regards,
Steffen,
 (probably the most hardcore TAP user on earth)
-- 
Steffen Schwigon <[email-masked]>

---

From: Andrew Rodland <[email-masked]>
Date: Fri, 19 Sep 2014 20:01:14 -0400

On Fri, Sep 19, 2014 at 4:16 AM, Ovid <[email-masked]> wrote:
> Hi Andrew,
>
> In that, we agreed that it was desirable that the determination of whether a
> line was junk/unknown wasn't context sensitive.
>
> I've tried searching for where I said that, but I can't find it. Can you
> provide me with the quote so I can understand where the confusion lies? I
> may very well be mistaken here.
>

=== begin quote (http://www.ietf.org/mail-archive/web/tap/current/msg00480.html)

That's a very annoying edge case you have there, but I would argue
that anything which *could* be parsed as a TAP line MUST be parsed as
such, thus rendering the above invalid. Otherwise, we'd have no idea
if you were spitting out bad TAP or something else was spitting out
junk which happened to look like TAP.

Thus:

* If something *can* be parsed as a TAP line, do so, even if it
renders the TAP document invalid.
* If something *can't* be parsed as a TAP line, ignore it.

=== end quote

> I fully admit that subtests were never considered in the original
> formulation of junk lines, but I think my description of this as being valid
> and the eight space indented line being an unknown (and therefore discarded)
> line is correct:
>
>> ok 1 - foo
> The reasoning works like this: historically it has always been the case that
> ok/not ok status is bound to the beginning of the string and even a single
> space in front of ok/not ok invalid. With a subtest, we make an exception if
> an only if it is indented exactly four spaces (in the context of the
> containing scope) and is parseable as TAP. Otherwise, writing a parser could
> be a nightmare. Consider:
>
>> ok
>> not ok
>> ok
>> 1..3
>
> Most people aren't aware of this, but that's a perfectly valid (but failing)
> TAP stream. While it's not recommended, the test numbers have always been
> optional. Here's the relevant line from the TAP grammar:
>
> Because TAP is designed to be both human-readable and computer-parseable,
> there exists the unfortunate problem that sometimes it's just a touch harder
> to parse than normal and some of the plans for it might make life more
> difficult. For example, if we followed your suggestion, we would then have
> this scenario:
>
>> ok
>>        ok this is working now
>> ok bar passed
>> 1..2
>
> As currently described (by my understanding), the above is valid because the
> "ok this is working now" line is indented too far and thus discarded. This
> makes parsing much simpler. However, you argue that the above is invalid
> because the "ok" line is valid TAP in an invalid context. That makes sense,
> but consider the plans for structured diagnostics:
>
>> not ok # TODO darn it!
>>     ---
>>     file     : t/foo.t
> Parsers are not required to parse structured diagnostics (this allows older
> parsers to ignore them), but that last line of the comment above could be
> considered valid TAP in an invalid context and result in a parse error, even
> though the above is perfectly fine.
>

Yes, I see the conflict here. I could suggest that structured
diagnostics be indented *two* spaces relative to the test that they
annotate, which would eliminate any potential for confusion since a
valid test is going to be indented 4n spaces. I'm not sure if that's
too esoteric.

---

From: Leon Timmermans <[email-masked]>
Date: Sun, 21 Sep 2014 00:09:48 +0200

On Wed, Sep 17, 2014 at 9:08 PM, Ovid <[email-masked]> wrote:

> A few notes on this.
>
> For subtests, they are indented four spaces. Always four spaces. If they
> are indented eight spaces, you know they're a subtest of a subtest
> (assuming there's an intervening subtest. That means this is valid TAP, but
> there is no subtest because it's indented too far:
>
> That is very useful to know, makes life for the consumer much easier.

> The "ok 1 - bar" line would be discarded as "unknown".

I think I agree, though one could argue that's an unknown subline of a
subtest.

> So if, in a stream of TAP, you find something which would be valid tap if
> it were unindented by four spaces, it's a subtest. The trailing,
> unindented summary line must mirror the status of the subtest. Thus, the
> following is invalid:
>
> ok 1 - foo
> Currently, Test::Harness <http://search.cpan.org/dist/Test-Harness/>
> treats the above as passing when it should not. I believe Schwern filed a
> bug for that. I know Andy Armstrong and I both took a quick swing at
> extending Test::Harness but discovered that it was more difficult than it
> appeared.
>

I'm currently giving a parse error in that condition :-)

As for the YAMLish structured diagnostics, those were never official, as
> far as I recall. There were several disputes, one of which was whether or
> not we should use YAML or JSON (I recall even XML being mentioned). Some
> argued that any structured diagnostic should be allowed, but I reject that
> firmly because that makes life hell for the parser. If you asked me what my
> stance is today, I would think that structured diagnostics should valid
> JSON, but that raises questions about whitespace issues, UTF-8, and so on.
> If we could agree on the format, then we'd need to agree on the keys in the
> structured diagnostic dictionary (and how to extend them).
>

Deciding on the keys is bikeshedding, deciding on the format decidedly
isn't. To be honest I'm not even sure what would be good or bad at this
stage, though I can agree with not accepting every format and their cousin.

Leon

---

From: Leon Timmermans <[email-masked]>
Date: Sun, 21 Sep 2014 01:22:03 +0200

> as additional additional bonus, already evidently seems to be hard to

I'd agree the lack of a clear opener is unfortunate. Parsing it requires
one to maintain more state than desirable.

> separate test files, respective TAP streams.

I can see the use for it, but it certainly adds complications.

> it for the last 6 years. I'm not exaggerating here.

I wasn't aware of anyone actually outputting YAML. Can you tell us more
about this?

Leon

---

From: Steffen Schwigon <[email-masked]>
Date: Mon, 22 Sep 2014 10:19:33 +0200

Leon Timmermans <[email-masked]> writes:
> On Fri, Sep 19, 2014 at 5:59 PM, Steffen Schwigon <[email-masked]> wrote:
> I wasn't aware of anyone actually outputting YAML. Can you tell us more about this?

With great pleasure. :-) Here we go:

The Tapper test infrastructure (http://tapper-testing.org/) consumes it
with the help of TAP::DOM, Data::DPath (and a handful marines), and it's
generated for embedded benchmark data in projects like Perl::Formance
(http://perlformance.net/), general supporting data in libs like
"bash-test-utils"
(https://github.com/tapper/Tapper-autoreport/blob/master/bash-test-utils.md)
and similar, but less public projects. In general we use it in a
restricted world like OS testing with only shell/printf/printk
available.

These projects might not look very active but that's only lack of github
pushes due to lack of tuits (usually just once a year during the Perl QA
hackathons). They are in very heavy use.

Slide decks (bit old but still valid):
 - http://tapper-testing.org/docs/yapc_eu_2011_tapjuggling.pdf - pg. 23, 37, 41
 - http://tapper-testing.org/docs/linuxcon_eu_2011_linux_and_virtualization_testing_with_tapper.pdf  pg. 35, 43

Two details make it harder to generate JSON than YAMLish:
 - on generating: JSON is oversensitive to trailing comma before closing brackets/braces.
   If you generate it from templates, like Template-Toolkit, you go mental.
 - on consuming: JSON defines Javascript-specific thingies, in particular true/false.
   Sounds minor, but makes it hard in restricted worlds like OS testing.

Kind regards,
Steffen

---

From: Leon Timmermans <[email-masked]>
Date: Tue, 23 Sep 2014 23:52:20 +0200

On Mon, Sep 22, 2014 at 10:19 AM, Steffen Schwigon <[email-masked]>
wrote:

> (
> https://github.com/tapper/Tapper-autoreport/blob/master/bash-test-utils.md
> )
>  -
> http://tapper-testing.org/docs/linuxcon_eu_2011_linux_and_virtualization_testing_with_tapper.pdf
> pg. 35, 43
>

Right. I see in your docs you're using two spaces (like the website
suggests). I'd really like to pin that down, unlike the \s+ on the website;
that's fuzzier than I like.

How do you visualize the data?

>    If you generate it from templates, like Template-Toolkit, you go mental.

Interesting. Never thought about it like that.

Leon