Why can't a program count its own tests

From Perl QA

Every once in a while someone asks, "Why not have the test program count up its own tests and set the plan number automatically?" Sounds like just a SMOP, right? Identify all possible test functions and look for them. No, its impossible. Not "impossible" meaning "difficult", "impossible" meaning "impossible. Its a variant of the halting problem which is the bane of many ideas which involve examining the code to figure out what it does.

Before I begin, the proper solution to this problem is to simply have no plan at all and use "no_plan". The test numbers are little but petty bookkeeping at this point, a relic of a time when we wrote print "ok 1\n" by hand and wanted to test tricky things like code execution order. There's better and more direct ways to do that (always have been).

Let's ignore for a moment the difficulties involved in identifying all possible test functions (or that someone might just write print "ok\n"). Consider the following.

   use Test::More;
   
   while(rand < .5) {
       pass;
   }

How many tests will that run? You can't know. Could be one. Could be none. Could be infinite!

Now you're saying, "sure, that's totally contrived. Nobody would ever do anything like that!" Ok, next.

   use Test::More;
    
   ok open(FILE, "test.data");
   
   while(my $line = <FILE>) {
       like $line, qr/poop is good food/;
   }

How many times will that run? What if its reading from a socket? Still not concrete enough for you? How about this:

   use Test::More;
   use Test::Pod;
   
   my @modules = find(sub { /\.pm$/ }, "lib");
   
   for my $module (@modules) {
       require_ok $file;
       pod_file_ok $file;
   }

How about a tricky situation like... a loop!

   until( $amount <= 0 ) {
       pay_some_more_bills($amount);
       like $amount, qr{^-?\d+\.\d{2}$};
   }

How about Test::NoWarnings?

   use Test::NoWarnings;

How many tests are there? Will you hard code everything that might do that?

I hope that answers the question.

[edit] But, but, but...!

[edit] What if we just write something that works for simple tests?

You wind up with something that can only handle things so simplistic you might as well just run the test and hard code the resulting number. Its the more complex situations where an automatic planner would have value.

And how do you know you're running it on "simple" code? If you can't trust the results of your automatic planner you need a human to double check the work, and then what's the point?