Perl Testing: Testing Mu in parallel with TAP::Harness

In the past year and a bit, I have started to experiment with writing tests in Perl. If you don't know about writing tests in Perl, an excellent place to start is Test::Tutorial. It introduces the concept and how to use some of Test::More.

My first project that I wrote tests for was dubbed "ThreadBoard". It was supposed to be a minimalist plugin-based threaded forum written in object orientated Perl that supported many different databases and generally followed good practice.

I started off with writing test code right away, which helped me figure out how to design the object orientated interface. I used Test::More along with some Perl code to run every test script. It worked fairly well with only a few test cases. This project didn't amount to anything, but I was able to reuse the code into Mu's forums.

Fall 2007. I decide that Mu is getting large enough that some test code would be nice to have. I start writing code like previously and end up with a hundred or so tests that hit 80% of the code. As it progresses, it becomes too hard to maintain the test suite, and I eventually give it up. Why? Mu had to use a database, so most actions had consequences that would affect test cases later on. Worse, I let each test case build upon previous results.

Another issue was the lack of speed. It took over 90 seconds to run the whole test suite, and you couldn't just run one test over and over, since each case needed all previous changes to be correct. This made testing very frustrating and hard to mentally keep track of what the condition was at a given point in the test cases.

This winter, I began to revisit the situation. My new Thinkpad's Core 2 Duo processor was much faster than my old Pentium 4, even without using the second core. But the second core opened up a new way to make it a faster: Run two tests at the same time.

However, this has a problem. All those old test cases would be difficult to convert to run in parallel. So I threw them out. I then decided to make the new cases able to run in parallel, which means that the other tests results can't affect the current one. This also means that each test file is easy to absorb in my mind: I get a clean slate at the beginning of each file.

The first challenge was running stuff in parallel. I used TAP::Harness in my test_all.pl file (maybe someday I'll use make test instead). A harness generally looks something like this:

use TAP::Harness;
my $harness = TAP::Harness->new({timer => 1, lib => '../lib', verbosity => 1, color => 1});
$harness->runtests(glob("*.t"));

Great. Now it runs tests. But, they're still sequential. So I make a small change:
use TAP::Harness;
my $harness = TAP::Harness->new({timer => 1, lib => '../lib', verbosity => 1, color => 1, jobs => 2});
$harness->runtests(glob("*.t"));
This is mentioned in the section about aggregate_tests rather than the new method for TAP::Harness.

My tests now run in parallel. But I need to run a small piece of code to clean out the databases first. I decided to add it as its own test case, and use some more complicated code:
use TAP::Harness;
my %main_args = (timer => 1, lib => '../lib', verbosity => 1, color => 1);
my $seq_harness = TAP::Harness->new({%main_args});
my $par_harness = TAP::Harness->new({%main_args, jobs => 2});

my $agg = TAP::Parser::Aggregator->new;
$agg->start;
$seq_harness->aggregate_tests($agg, 'init_test.pl');
$par_harness->aggregate_tests($agg, glob('*.t'));
$agg->stop;

Perfect. Now I have sequential tests running, then parallel tests. I can switch between the two modes later if I need by adding more cases here.

This post has become fairly long, so I'll stop here. Next time, I'll talk about code coverage (Devel::Cover), documentation coverage (Pod::Coverage), and how I'm trying to make adding documentation easier (using the PPI module).