Glassbox Testing

From Perl QA
Jump to: navigation, search

(This is a piece I wrote on the advantages and traps of glassbox testing on the PDX.pm mailing list which I thought was worth archiving. --Schwern)

Josh Heumann wrote:
> While we're talking about testing, what's the general feeling on testing
> protected and private methods?

If you feel it will be easier to test functionality via private methods, do it. There's even a name for it: glassbox testing.

The trouble with pure blackbox testing is that in order to get at a specific piece of functionality you might have to go through a whole lot of code to get to the one piece you care about. When the test fails you don't know if its the piece you're testing or something in between. Glassbox allows you to drill down and test a specific function which does a specific thing. Once you're sure that unit (see, its unit testing) is working reliably then you can move on to test the layer above it and then above that and so on.

An example just last week. I'm working with a big hairy do_search() sort of function which does all the magical voodoo necessary to write and run the SQL to perform searches on a particular web site. I refactored the code to take a glob style search (ie. foo*) and make it into a LIKE style search (ie. foo%) into its own private method, _glob2like().

Now, doing this blackbox I would have had to...

A) Mock up some test search data in the database interesting enough to exercise my wildcard function. This is not trivial in and of itself. Worse there's the additional complication that there are no tests for the shared search at all so I'd have to write that whole framework.

B) Figure out the right arguments to pass into do_search() (it was not a trivial function)

C) Divine from the results of the search whether or not my wildcard worked.

Repeat for each interesting glob input I want to test. And if a test fails I have to dig in to find out if its because of my new methods or because they're being called wrong.

Ugh.

So instead I just tested the private methods directly which is trivial.

 is Class->_glob2like("foo*"), "foo%";

Once I knew _glob2like() was reliable (and it initially wasn't) I can go ahead and write tests for the larger search routine without the extra complexity of worrying about testing _glob2like(). I already know its reliable.

The additional advantage of glassbox testing is it encourages breaking down huge_hairy_functions() into small_sharp_functions(). In the example above, do_search() is huge and hairy and contains a lot of ifs and elsifs. I could have just put the code for _glob2like() inline in the function, in fact initially it was, but that would have been really hard to test. So I broke it out into a nice, sharp, easily testable, easily understood function and avoided making do_search() even more complicated. I have also now created the opportunity for someone in the future who needs the same thing to elevate _glob2like(), complete with its own independent tests, into a public SQL utility method.


The disadvantage of glassbox is a failure in a glassbox test does not mean a real bug in the program. Maybe the guts were just redesigned. Too many glassbox tests can make the test suite very twitchy about any sort of internal redesign. The important thing is to make it obvious what is a glassbox and what is a blackbox test. Usually marking your non-public methods in some way (ie. _this_is_not_public()) does the trick.

A trap of glassbox testing is you might never get around to testing it from the blackbox perspective. You have a lot of glassbox tests testing the individual units but nothing testing that it actually works once its all put together. All those glassbox tests lend themselves to a false sense of completeness. Its a trap, not a disadvantage. The glassbox tests are to your advantage when it comes to blackbox testing because you have less to test via the more complicated blackbox.

Personal tools