TAP::Parser

From Perl QA

Jump to: navigation, search

A TAP parser being written by Ovid, Andy Armstrong and Eric Wilhelm. TAP::Parser used to be called TAPx::Parser.

The current development version can be obtained from the subversion repository.

The latest release can be obtained from the CPAN.

The TAP::Parser developers hang out on this mailing list.

As of March 11, 2007, here's the current roadmap:

[edit] DONE

[edit] 0.51

'execrc' file now allows 'regex' matches for tests.
rename 'TAPx' --> 'TAP'
Reimplemented the parse logic of TAP::Parser as a state machine.
Removed various ad-hoc state variables from TAP::Parser and moved their logic into the state machine.
Removed now-unused is_first / is_last methods from Iterator and simplified remaining logic to suit.
Removed now-redundant t/140-varsource.t.
Implemented TAP version syntax.

[edit] 0.50_07 5 March 2007

Fixed bug where we erroneously checked the test number instead of number of tests run to determine if we've run more tests than we planned.
Add a --directives switch to 'runtests' which only shows test results with directives (such as 'TODO' or 'SKIP').
Removed some dead code from TAPx::Parser.
Added color support for Windows using Win32::Console.
Made Color::failure_output reset colors before printing the trailing newline.
Corrected some issues with the 'runtests' docs and removed some performance notes which no longer seem accurate.
Fixed bug whereby if tests without file extensions were included then the spacing of the result leaders would be off.
execrc file is now a YAML file.
Removed white background on the test failures. It was too garish for me. Just more proof that we need better ways of overriding color support.
Started work on TAPx::Harness::Compatible. Right now it's mainly just a direct lift of Test::Harness to make sure the tests work.
Commented out use Data::Dumper::Simple in T::Harness.pm - it's not a core module.
Added next_raw to TAPx::Parser::Iterator which skips any fixes for quirky TAP that are implemented by next. Used to support TAPx::Harness::Compatible::Iterator
Applied our version number to all T::H::Compatible modules
Removed T::H::C::Assert. It's documented as being private to Test::Harness and we're not going to need it.
Refactored runtests to call aggregate_tests to expose the interface we need for the compatibility layer.
Make it possible to pass an end time to summary so that it needn't be called immediately after the tests complete.
Moved callback handling into TAPx::Base and altered TAPx::Parser to use it.
Made TAPx::Harness into a subclass of TAPx::Base and implemented made_parser callback.
Moved the dispatch of callbacks out of run and into next so that they're called when TAPx::Harness iterates through the results.
Implemented PERL_TEST_HARNESS_DUMP_TAP which names a directory into which the raw TAP of any tests run via TAPx::Harness will be written.
Rewrote the TAPx::Grammar->tokenize method to return a TAPx::Parser::Result object. Code is much cleaner now.
Moved the official grammar from TAPx::Parser to TAPx::Parser::Grammar, provided a link and updated the grammar.
Fixed bug where a properly escaped '# TODO' line in a test description would still be reported as a TODO test.
Added patches/ExtUtils-MakeMaker-6.31.patch - a patch against EUMM that makes test_harness use TAPx::Harness instead of Test::Harness if PERL_EUMM_USE_TAPX is true and TAPx::Harness is installed. In other words cause 'make test' for EUMM based models to use TAPx::Harness.
Added support for timer option to TAPx::Harness which causes the elapsed time for each test to be displayed.
Setup tapx-dev@hexten.net mailing list.
Fixed accumulating @$exec bug in TAPx::Harness.
Made runtests pass '--exec' option as an array.
(#24679) TAPx::Harness now reports failure for tests that die after completing all subtests.
Added in_todo attribute on TAPx::Parser which is true while the most recently seen test was a TODO.
(#24728) TAPx::Harness now supresses diagnostics from failed TODOs. Not sure if the semantics of this are correct yet.

[edit] 0.50_06 18 January 2007

Fixed doc typo in examples/README (rt.cpan.org #24409)
Colored test output is now the default for 'runtests' unless you're running under windows or -t STDOUT is false. (rt.cpan.org #24310)
Removed the .t extension from t/source_tests/*.t since those are 'test tests' which caused false negatives when running recursive tests. [Adrian Howard]
Somewhere along the way, the exit status started working again. Go figure.
Factored color output so that disabling it under Windows is cleaner.
Added explicit switch to :crlf layer after open3 under Windows. open3 defaults to raw mode resulting in spurious \r characters input parsed input.
Made Iterator do an explicit wait for subprocess termination. Needed to get process status correctly on Windows.
Fixed bug which didn't allow t/010-regression.t to be run directly via Perl unless you specified Perl's full path.
Removed SIG{CHLD} handler (which we shouldn't need I think because we explicitly waitpid) and made binmode ':crlf' conditional on IS_WIN32. On Mac OS these two things combined to expose a problem which meant that output from test scripts was sometimes lost.
Made t/110-source.t use File::Spec->catfile to build path to test script.
Made Iterator::FH init is_first, is_last to 0 rather than undef for consistency with array iterator.
Added t/120-varsource.t to test is_first and is_last semantics over files with small numbers of lines.
Added check for valid callback keys.
Added t/130-results.t for Result classes.

[edit] 0.50_05 15 January 2007

Removed debugging code accidentally left in bin/runtests.
Removed 'local $/ = ...' from the iterator. Hopefully that will fix the line ending bug, but I don't know about the wstat problem.

[edit] 0.50_04 14 January 2007

BACKWARDS IMCOMPATIBLE: Renamed all '::Results' classes to '::Result' because they represent a single result.
Fixed bug where piping would break verbose output.
IPC::Open3::open3 now takes a @command list rather than a $command string. This should make it work under Windows.
Added 'stdout_sterr' sample test back to regression tests. IPC::Open3 appears to make it work.
Bug fix: don't print 'All tests successful' if no tests are run.
Refactored 'runtests' to make it a bit easier to follow.
Bug fix: Junk and comments now allowed before a leading plan.
HARNESS_ACTIVE and HARNESS_VERSION environment variables now set.
Renamed 'problems' in TAPx::Parser and TAPx::Aggregator to 'has_problems'.

[edit] 0.50_03 08 January 2007

Fixed bug where '-q' or '-Q' with colored tests weren't suppressing all information.
Fixed an annoying MANIFEST nit.
Made '-h' for runtests now report help. Using a new harness requires the full --harness switch.
Added 'problems' method to TAPx::Parser and TAPx::Parser::Aggregator.
Deprecatd 'todo_failed' in favor of 'todo_passed'
Add -I switch to runtests.
Fixed runtests doc nit (smylers)
Removed TAPx::Parser::Builder.
A few more POD nits taken care of.
Completely removed all traces of C<--merge> as IPC::Open3 seems to be working.
Moved the tprove* examples to examples/bin in hopes of them no longer showing up in CPAN's docs.
Made the 'unexpectedly succeeded' message clearer (Adam Kennedy)

[edit] 0.50_02 06 January 2007

Added some files I left out of the manifest (reported by Florian Ragwitz).
Added strict to Makefile.PL and changed @PROGRAM to @program (reported Florian Ragwitz).

[edit] 0.50_01 06 January 2007

Added a new example which shows to how test Perl, Ruby, and URLs all at the same time using 'execrc' files.
Fixed the diagnostic format mangling bug.
We no longer override Test::Builder to merge streams. Instead, we go ahead and use IPC::Open3. It remains to be seen whether or not this is a good idea.
Fixed vms nit: for failing tests, vms often has the 'not' on a line by itself.
Fixed bugs where unplanned tests were not reporting as a failure (test number greater than tests planned).
TAPx::Parser constructor can now take an 'exec' option to tell it what to execute to create the stream (huge performance boost).
Added TAPx::Parser::Source. This allows us to run tests in just about any programming language.
Renamed the filename() method to source() in TAPx::Parser::Source::Perl.
We now cache the @INC values found for TAPx::Parser::Source::Perl.
Added two test harnesses, TAPx::Harness and TAPx::Harness::Color.
Removed references to manual stream construction from TAPx::Parser documentation. Users should not (usually) need to worry about streams.
Added bin/runtests utility. This is very similar to 'prove'.
Renumbered tests to make it easier to add new ones.
Corrected some minor documentation nits.
Makefile.PL is no longer auto-generated (it's built by hand).
Fixed regression test bug where driving tests through the harness I'm testing caused things to break.
BUG: exit() values are now broken. I don't know how to capture them with IPC::Open3. However, since no one appears to be using them, this might not be an issue.

[edit] 0.41 12 December 2006

Fixed (?) 10-regression.t test which failed on Windows. Removed the segfault test as it has no meaning on Windows. Reported by PSINNOTT <link@redbrick.dcu.ie> and fix recommended by Schwern based on his Test::Harness experience. (rt.cpan.org #21624)

[edit] 0.40 05 December 2006

Removed TAPx::Parser::Streamed and folded its functionality into TAPx::Parser.
Fixed bug where sometimes is_good_plan() would return a false positive (exposed by refactoring).
A number of tiny performance enhancements.

[edit] 0.33 22 September 2006

OK, I'm getting ticked off by some of the comments on Perl-QA so I rushed this out the door and broke it :( I'm backing out one test and slowing down a bit.

[edit] 0.32 22 September 2006

Applied patch from Schwern which fixed the Builder package name (TAPx:: instead of TAPX:: -- stupid case-insensitive package names!). (rt.cpan.org #21605)

[edit] 0.31 21 September 2006

Fixed bug where Carp::croak without parens could cause Perl to fail to compile on some platforms. [Andreas J. Koenig]
Eliminated the non-portable redirect of STDERR to STDOUT (2>&1) and fixed the synchronization issue. This involves overridding Test::Builder::failure_output() in a very sneaky way. I may have to back this out.
Renamed boolean methods to begin with 'is_'. The methods they replace are documented, deprecated, and will not be removed prior to version 1.00.

[edit] 0.30 17 September 2006

Fixed bug where no output would still claim to have a good plan.
Fixed bug where no output would cause parser to die.
Fixed bug where failing to specify a plan would be two parse errors instead of one.
Fixed bug where a correct plan count in an incorrect place would still report as a 'good_plan'.
Fixed bug where comments could accidently be misparsed as directives.
Eliminated testing of internal structure of result objects. The other tests cover this.
Allow hash marks in descriptions. This was causing a problem because many test suites (Regexp::Common and Perl core) allowed them to exist.
Added support for SKIP directives in plans.
Did some work simplifying &TAPx::Parser::_initialize. It's not great, but it's better than it was.
TODO tests now always pass, regardless of actual_passed status.
Removed 'use warnings' and now use -w
'switches' may now be passed to the TAPx::Parser constructor.
Added 'exit' status.
Added 'wait' status.
Eliminated 'use base'. This is part of the plan to make TAPx::Parser compatible with older versions of Perl.
Added 'source' key to the TAPx::Parser constructor. Making new parsers is now much easier.
Renamed iterator first() and last() methods to is_first() and is_last(). Credit: Aristotle.
Planned tests != tests run is now a parse error. It was really stupid of me not to do that in the first place.
Added massive regression test suite in t/100-regression.t
Updated the grammar to show that comments are allowed.
Comments are now permitted after an ending plan.

[edit] 0.22 13 September 2006

Removed buggy support for multi-line chunks from streams. If your streams or iterators return anything but single lines, this is a bug.
Fixed bug whereby blank lines in TAP would confuse the parser. Reported by Torsten Schoenfeld.
Added first() and last() methods to the iterator.
TAPx::Parser::Source::Perl now has a 'switches' method which allows switches to be passed to the perl executable running the test file. This allows tprove to accept a '-l' argument to force lib/ to be included in Perl's @INC.

[edit] 0.21 8 September 2006

Included experimental GTK interface written by Torsten Schoenfeld.
Fixed bad docs in examples/tprove_color
Applied patch from Shlomi Fish fixing bug where runs from one stream could leak into another when bailing out. (rt.cpan.org #21379)
Fixed some typos in the POD.
Corrected the grammar to allow for a plan of "1..0" (infinite stream).
Started to add proper acknowledgements.

[edit] 0.20 2 September 2006

Fixed bug reported by GEOFFR. When no tap output was found, an "Unitialized value" warning occurred. (rt.cpan.org #21205)
Updated tprove to now report a test failure when no tap output found.
Removed examples/tprove_color2 as tprove_color now works.
Vastly improved callback system and updated the docs for how to use them.
Changed TAPx::Parser::Source::Perl to use Symbol::gensym() instead of a hard-to-guess filehandle name.

[edit] 0.12 30 July 2006

Added a test colorization script
Callback support added.
Added TAPx::Parser::Source::Perl.
Added TAPx::Parser::Aggregator.
Added version numbers to all classes.
Added 'todo_failed' test result and parser.
00-load.t now loads all classes instead of having individual tests load their supporting classes.
Changed $parser->results to $parser->next

[edit] 0.11 25 July, 2006

Renamed is_skip and is_todo to has_skip and has_todo. Much less confusing since a result responding true to those also responded true to is_test.
Added simplistic bin/tprove to run tests. Much harder than I thought and much code stolen from Test::Harness.
Modified stolen iterator to fix a bug with stream handling when extra newlines were encountered.
Added TAPx::Parser::Iterator (stolen from Test::Harness::Iterator)
Normalized internal structure of result objects.
All tokens now have a 'type' key. This greatly simplifies internals.
Copied much result POD info into the main docs.
Corrected the bug report URLs.
Minor updates to the grammar listed in the POD.

[edit] 0.10 23 July, 2006

Oh my Larry, we gots docs!
_parse and _tap are now private methods.
Stream support has been added.
Moved the grammar into its own class.
Pulled remaining parser functionality out of lexer.
Added type() method to Results().
Parse errors no longer croak(). Instead, they are available through the parse_errors() method.
Added good_plan() method.
tests_planned != tests_run is no longer a parse error.
Renamed test_count() to tests_run().
Renamed num_tests() to tests_planned().

[edit] 0.03 17 July, 2006

'Bail out!' is now handled.
The parser is now data driven, thus skipping a huge if/else chain
We now track all TODOs, SKIPs, passes and fails by test number.
Removed all non-core modules.
Store original line for each TAP line. Available through $result->raw().
Renamed test is_ok() to passed() and added actual_passed(). The former method takes into account TODO tests and the latter returns the actual pass/fail status.
Fixed a bug where SKIP tests would not be identified correctly.

[edit] TODO

Figure out the namespace.

Allow description transformations to be applied. For example:

$description =~ s/\A\s*-\s*//; # remove leading '- '

Improve error reporting

Make it easier to find tests which unexpectedly succeed (TODO tests which pass when they shouldn't)

[edit] QUESTIONS

skipped lines should always be OK? For example:

not ok 3 - we blew it! # SKIP should never be "not ok"'

This will probably result in a parse error.

Retrieved from "http://perl-qa.hexten.net/wiki/index.php/TAP::Parser"

Category: TAP