TAP::Harness

From Perl QA
Jump to: navigation, search

TAP::Harness was supposed to be a completely new TAP framework, written by Schwern. However, development on it has stalled, and it was phased out in favor of TAPx::Parser and TAPx::Harness:

<rindolf>    Schwern: any progress with TAP::Harness?
<Schwern>    Ovid has taken that over with TAPx::Parser.
             They're talking about gutting Test::Harness 
             and using it as the engine making TH 3.0
<rindolf>    Schwern: I see.

Nevertheless, this page may still be of interest for ideas or insights.

The latest TAP::Harness code is available in its repository: Repository.


Contents

Design Philosophy

TAP::Harness is designed as a clean break with Test::Harness yet filling the same niche. It will break down the handling of TAP into several distinct stages each represented by its own object.

  1. Running the TAP source and streaming the output (ie. executing foo.t)
  2. Lexing the TAP into tokens. (test line, diagnostic, etc...)
  3. Parsing the TAP tokens in context.
  4. Aggregating the TAP results. (how many passed, how many failed, etc...)
  5. Displaying the TAP output.

Very roughly, it flows like this...

     Source
       |
   (raw lines)
       |
       v
     Lexer
       |
    (tokens)
       |
       v
     Parser
       |
    (events)
       |
   Aggregator
       |
(events w/ context)
       |
       v
    Display

Each point in the process is represented by an object or group of objects. You will be able to replace that object with your own to customize things. Typically this will be adding new Sources and/or your own Display.


Design Goals

Replace Test::Harness
TAP::Harness should fill every niche Test::Harness does.
No external dependencies
Because the goal is to go into the core it can have no dependencies on non-core modules. Module::Compile may mitigate this.
Modular
Each piece should be replacable by a work-alike.
Handle TAP upgrades
Allow users to control the display of TAP results without knowledge of TAP parsing details.
Allow users to write plugins to take TAP from any source without knowledge of TAP parsing details.

Use Cases

Fancy GUI and HTML outputs

Multiple, non-Perl TAP sources

See TAP::Harness::Source below.

"enhanced" TAP output

For example, colorized test output.

Smoothly handle TAP extensions.

Probably by providing a separate parsing object for each version of TAP.

Adapting other testing protocols to TAP.

Probably by writing a Source which translates from the foreign protocol to TAP.

Parallel test runs

I'd like to be able to specify parallel test runs. Specifically, I'd like to be able to send tests as a set of groups, with instructions on how the tests are to be parallelised. I already have a prototype, hacked from Test::Harness.

  1. Send tests as groups
  2. Tests within a group run in series (in traditional order)
  3. Groups run in parallel
  4. have sync points. All groups before a sync point must complete before any groups after a sync point can start.

This let me run many of the core tests in parallel, but still allowed the tests that need to run in order to run in order. This level of control let me run the core tests as follows:

t/base/*.t
one by one (each test is its own group, sync point between each group)
t/comp/*.t t/cmd/*.t t/run/*.t t/io/*.t t/op/*.t
tests in each directory in sequence, but the 5 directories in parallel wait for all these to finish
t/uni/*.t, all tests in ext and lib
tests in a the same directory ran in series. All directories could run in parallel

With this I can max out a multi CPU (or multi core) box. (Or at least stand a fighting chance, although tests in t/op/*.t in the core need some issues fixing before they can run in parallel with each other. I suspect that their names for temporary files are unimaginative and therefore clash)

Components

TAP::Harness::Source

A TAP source is anything which can be made to produce a stream of TAP. A TAP::Harness::Source takes a raw source (such as a filename) and does whatever is necessary to make it produce a stream of TAP (such as run it as a Perl program).

TAP::Harness::Source is intended as an adaptor around the many possible sources of TAP. Each source can be handed a raw source and asked whether or not it can handle it.


TAP::Harness::Source::String

This would take a reference to a string and chop it up into lines of TAP. Useful for testing or in situations where you don't have to stream.

TAP::Harness::Source::PerlDotT

Emulates prove and Test::Harness's behavior by running a Perl program with the appropriate @INC.

TAP::Harness::Source::HTMLValidator

This would take a .html file and run it through an HTML validator. The Source would produce TAP representing what parts of the validation it passed or failed.

TAP::Harness::Source::HTTP

An HTTP URI would be fetched and the resulting page streamed as TAP.

TAP::Harness::Source::Executable

The raw source would simply be executed and STDOUT streamed as TAP.


TAP::Harness::Runner

The runner is a factory which selects which TAP::Harness::Source to use for a given source by asking each source in turn if it can handle the raw source. The list of sources is pluggable.


TAP::Harness::Lexer

This reads in raw lines from a TAP::Harness::Source and lexes them into TAP::Harness::Tokens. No semantic parsing is done at this time. For example, the test numbers are not checked to see if they are in sequence, this is the parser's responsibility.


TAP::Harness::Token

Token types include:

  • Test
  • Header
  • Diagnostic
  • Unknown
  • Error

Headers common to all tokens include:

  • Raw line
  • Error

TAP::Harness::Parser

This takes a stream of tokens from the lexer and assigns semantic meaning to them. For example, checking if the test numbers are all in order or if the wrong number of tests are run.

It may also collect the results of a single test run. How many tests were run. How many failed. How many passed.

Details of this stage are a little fuzzy.

TAP::Harness::Aggregator

The aggregator runs multiple parsers and collects their information together. This is the summarization of the results of all TAP sources parsed. How many sources were run. Which passed. Which failed.

Details of this stage are a little fuzzy.

TAP::Harness::Display

The display is handed a stream of events and has access to the tallies of the currently running source and the overall results of all sources. The display can then do whatever it wants with these results. It can spit out XML. It can drive a GUI. It can write out to a file. Whatever.

TAP::Harness::Event

An event handed to the Display. Each event will carry this sort of information:

  • Details about this particular event (pass/fail, test number, description, etc...).
  • The original TAP line.
  • Its context. (source, uri, file, line number, etc...)
  • Any TAP format error associated with this event. (test number out of order)
  • The summary for the current source.
  • The summary for all sources so far.


TAP::Harness

This pulls all the individual objects together and "turns the crank" to make the process go.


FAQ

Why call it TAP::Harness?

"TAP" focuses it on the fact that this is about the Test Anything Protocol, not just Perl's testing stuff. By not putting it in the Test:: namespace I hope to draw a stronger line between the responsibilities of things like Test::More (your test program) and TAP::Harness (the thing which runs and interprets your test program).

"Harness" is a link with its forerunner, Test::Harness.

What about Test::Harness?

Test::Harness remains its own thing.

At some point in the future Test::Harness will likely be gutted and turned into a thin wrapper around TAP::Harness. I'm not caring about this right now.


Is it going to use Test::Harness::Straps?

No. I will be stealing lots of code from Straps, but I will not be using Straps. Straps has too many design flaws. It tries to do too many parts of the TAP processing. It also doesn't do all the TAP processing leaving the Straps user to do some of that. And the callback system doesn't work very well.


Should I continue to work on <other TAP parser>?

Yes. While I am optimistic, I make no promises as to when TAP::Harness is going to be stable. So keep working on Test::Harness or Test::Run or whatever. Keep using Test::Harness::Straps.


Will TAP::Harness go into the core?

Probably at some point. I don't care right now.


Will I be able to do X with TAP::Harness?

The goal is to encompass the largest set of X. Another goal is to have the extender be able to focus on the display aspects and not the TAP parsing.


Will TAP::Harness include X extension to TAP?

No. TAP::Harness is using the current TAP spec from Test::Harness::TAP. Extending TAP is another problem.


Will I be able to test X (HTML, PHP, Javascript, Monkeys...) with TAP::Harness?

Yes. You will be able to write new "TAP source" plugins for whatever you want. As long as it winds up producing a stream of TAP at the end.


How can I help?

Provide use cases, what would you want to do with Test::Harness if you could? What are you doing with Straps? What features do other testing systems (JUnit, for example) have that you'd like to see in Perl? Pick apart the design.

Personal tools