Why can't a program count its own tests

From Perl QA
Jump to: navigation, search

Every once in a while someone asks, "Why not have the test program count up its own tests and set the plan number automatically?" Sounds like just a SMOP, right? Identify all possible test functions and look for them. No, its impossible. Not "impossible" meaning "difficult", "impossible" meaning "impossible. Its a variant of the halting problem which is the bane of many ideas which involve examining the code to figure out what it does.

Before I begin, the proper solution to this problem is to simply have no plan at all and use "no_plan". The test numbers are little but petty bookkeeping at this point, a relic of a time when we wrote print "ok 1\n" by hand and wanted to test tricky things like code execution order. There's better and more direct ways to do that (always have been).

Let's ignore for a moment the difficulties involved in identifying all possible test functions (or that someone might just write print "ok\n"). Consider the following.

   use Test::More;
   while(rand < .5) {

How many tests will that run? You can't know. Could be one. Could be none. Could be infinite!

Now you're saying, "sure, that's totally contrived. Nobody would ever do anything like that!" Ok, next.

   use Test::More;
   ok open(FILE, "test.data");
   while(my $line = <FILE>) {
       like $line, qr/poop is good food/;

How many times will that run? What if its reading from a socket? Still not concrete enough for you? How about this:

   use Test::More;
   use Test::Pod;
   my @modules = find(sub { /\.pm$/ }, "lib");
   for my $module (@modules) {
       require_ok $file;
       pod_file_ok $file;

How about a tricky situation like... a loop!

   until( $amount <= 0 ) {
       like $amount, qr{^-?\d+\.\d{2}$};

How about Test::NoWarnings?

   use Test::NoWarnings;

How many tests are there? Will you hard code everything that might do that?

I hope that answers the question.

But, but, but...!

What if we just write something that works for simple tests?

You wind up with something that can only handle things so simplistic you might as well just run the test and hard code the resulting number. Its the more complex situations where an automatic planner would have value.

And how do you know you're running it on "simple" code? If you can't trust the results of your automatic planner you need a human to double check the work, and then what's the point?

Other languages do it differently

In other languages that have this kind of unit-testing framework, they just count the tests as you go. Each time you run a test - a number is added to a counter. And then at the very end, the total amount of tests is displayed - together with the amount of failed tests and passed tests. Also the "ok 5" messages have no problem using that counter to just display the $counter+1 number each time they need to be displayed. But unfortunately, since this approach is not accepted, Test::Simple and Test::More and friends don't do this. And this article had to be written, instead of learning from other languages and fixing the problem once and for all.

Personal tools