Subject: [tap] W3C Evaluation and Report Language (EARL)

From: Christophe Strobbe <[email-masked]>
Date: Wed, 09 Nov 2011 14:23:18 +0100

Hello,

I am a participant of the W3C's Evaluation and Repair Tools Working 
Group (ERT WG), a working group that develops the Evaluation and 
Report Language (EARL) and a few related specifications. EARL's core 
use case is reporting the results of accessibility evaluations of 
websites (i.e. accessibility for persons with disabilities), but the 
language itself is generic, so it can also be used in other contexts. 
The language is based on RDF; the specification is available at 
<http://www.w3.org/TR/EARL10/>.

During our last call for comments, one of the reviewers asked the 
working group if EARL duplicates TAP's efforts, or vice versa. The 
working group thinks that this is not the case; we think that EARL 
could be an alternative report format for TAP if a TAP consumer could 
be written that produces EARL. For this reason, we thought it would 
be interesting to contact you and to make sure we are aware of each 
other's work.

Shadi Abou-Zahra, in CC, is the working group's chair and W3C staff contact.

Best regards,

Christophe

-- 
Christophe Strobbe
K.U.Leuven - Dept. of Electrical Engineering - SCD
Research Group on Document Architectures
Kasteelpark Arenberg 10 bus 2442
B-3001 Leuven-Heverlee
BELGIUM
tel: +32 16 32 85 51
http://www.docarch.be/
Twitter: @RabelaisA11y
---
Open source for accessibility: results from the AEGIS project 
www.aegis-project.eu
---
Please don't invite me to Facebook, Quechup or other "social 
networks". You may have agreed to their "privacy policy", but I haven't.

---

From: Ovid <[email-masked]>
Date: Wed, 9 Nov 2011 07:06:25 -0800 (PST)

Hi Christophe,

Thanks for dropping us a note.

Just from glancing through the EARL specs, it appears that there is some overlap, but the two are not congruent. A TAP producer would have a difficult time producing the requisite output for a few reasons.

1. You produce output we don't support natively in a TAP stream.
2. RDF documents are, well, documents, not streams.

We're working on a proposal which could address some of point #1, but as TAP allows for infinite streams (for example, repeatedly pinging a server to see that it's responsive), I don't think an infinite stream wouldn't be well-represented in a documented-oriented protocol.

As a side note, from glancing at http://trac.usefulinc.com/doap/wiki/ProjectGoals:

* We don't support "suspend[ing] more zombies in the row".
* I'm uncertain how you plan to extend the vocabulary for http://hepburnspringsaccommodation.webs.com
* Zombies are actually made of tasty, tasty spam (we eat them and before they eat us)

You might want someone to take a look at that page and clean up the spam :)

Cheers,
Ovid
--
Live and work overseas - http://overseas-exile.blogspot.com/
Buy the book           - http://www.oreilly.com/catalog/perlhks/
Tech blog              - http://blogs.perl.org/users/ovid/
Twitter                - http://twitter.com/OvidPerl/

>________________________________
>From: Christophe Strobbe <[email-masked]>
>To: [email-masked]
>Cc: Shadi Abou-Zahra <[email-masked]>
>Sent: Wednesday, 9 November 2011, 14:23
>Subject: [tap] W3C Evaluation and Report Language (EARL)
>
>Hello,
>
>I am a participant of the W3C's Evaluation and Repair Tools Working Group (ERT WG), a working group that develops the Evaluation and Report Language (EARL) and a few related specifications. EARL's core use case is reporting the results of accessibility evaluations of websites (i.e. accessibility for persons with disabilities), but the language itself is generic, so it can also be used in other contexts. The language is based on RDF; the specification is available at <http://www.w3.org/TR/EARL10/>.
>
>During our last call for comments, one of the reviewers asked the working group if EARL duplicates TAP's efforts, or vice versa. The working group thinks that this is not the case; we think that EARL could be an alternative report format for TAP if a TAP consumer could be written that produces EARL. For this reason, we thought it would be interesting to contact you and to make sure we are aware of each other's work.
>
>-- Christophe Strobbe
>Open source for accessibility: results from the AEGIS project www.aegis-project.eu
>---
>Please don't invite me to Facebook, Quechup or other "social networks". You may have agreed to their "privacy policy", but I haven't.
>
>_______________________________________________
>tap mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/tap
>
>
>

---

From: Steffen Schwigon <[email-masked]>
Date: Wed, 09 Nov 2011 20:25:04 +0000

Christophe Strobbe <[email-masked]> writes:
> language itself is generic, so it can also be used in other
> contexts. The language is based on RDF;
> […]
> be written that produces EARL. For this reason, we thought it would be
> interesting to contact you and to make sure we are aware of each
> other's work.

Thanks for sync'ing this back to us. I just skimmed through the specs
and it was indeed interesting. As far as I understand from my (very
short) skimming I think it's not that many duplication of effort as the
main difference is a philosophical one.

- EARL is similar to other W3C specs in respect to specifying a
  comprehensive snapshot of known existing topics. For example, it
  particularly covers all known HTTP methods (POST, GET, PUT, …). That
  enables it to build tools on top of it that sematically “know” what
  the document is about.

- TAP in contrast is about specifying test results, really just the
  *result* focus without hard specification of the tested topic, i.e., a
  single test has a “description”, so someone reading it knows what it
  is about but that part does not have a specification. 

  For instance, a test about a HTTP method could have any description
  from “POST” to “that strange other method that I never remember but
  always use when GET is not sufficient”.

See [1] for some related discussion of this aspect.

In this respect I think TAP is more like your RDF with some extensions
from EARL to describe test success.

That makes the use-cases of TAP and EARL a bit different: 

- TAP allows to be produced by anything simple without toolchain
  support, like embedded devices with nothing but a “print” function,
  but you can not *sematically* evaluate results.

- EARL seems to require more heavy toolchain support to produce but
  allows more semantic result evaluation.

Converting TAP to EARL is difficult. 
Converting EARL to TAP is easy.

On the evaluation of TAP I can point to TAP::DOM and Data::DPath, which
provide a more structured approach to evaluate test results, see my “TAP
Juggling” slides[2], page 30ff.

Kind regards,
Steffen

Footnotes: 
[1]  http://grokbase.com/p/perl.org/qa/2008/04/re-tap-l-user-supplied-yaml-diagnostic-keys-descriptive-version/11ymnpm2765ztojoinznq2lz5674

[2]  http://www.amd64.org/fileadmin/user_upload/pub/yapc_eu_2011_tapjuggling.pdf

-- 
Steffen Schwigon <[email-masked]>
Dresden Perl Mongers <http://dresden-pm.org/>

---

From: Leon Timmermans <[email-masked]>
Date: Wed, 9 Nov 2011 21:49:40 +0100

>  allows more semantic result evaluation.

I agree. To me, EARL makes sense in a context that heavy into semantic
web and triplestores, but the toolchain would be too heavy for a lot
of usecases.

> Converting EARL to TAP is easy.

Making valid EARL out of TAP shouldn't be difficult. Useful EARL on
the other hand

> Juggling” slides[2], page 30ff.

Interesting…

Leon