Parallel run-tests performance

The development of parallel run-tests.php is moving along. Unfortunately paid work is going to get in the way a bit for the next few weeks so things will move a little slowly.

I’m still thinking about the REDIRECTTEST implementation. I don’t think it’s that hard, the other thing that needs to be thought through is exactly how to avoid clashes between, for example, msql and mysqli tests. I think that the latter is best addressed by having a configuration file of test directories that have to be run in sequence. In a parallel run, one processor would be set to work through these while other tests were scheduled randomly to the remaining processors. Anyway – it needs some thought and the parallel execution code is the part I’m least familiar with.

The other thing that really needs to be done is some more extensive performance work. I have done a couple of runs on a dual core Mac, just to get some data points.

The tests in the timing benchmark (phpruntests/QA/QATimedBucket.tgz) are taken from the PHP development stream and are all tests under:



ctype date dom ereg fileinfo filter iconv json libxml pcre phar 
posix reflection session spl sqlite3 standard tokenizer xml 
xmlreader xmlwriter zlib

 

I executed these tests three times. Run 1 uses the current PHP development stream version of run-tests.php. Run 2 uses the parallel version of run-tests.php run in sequential mode and Run 3 uses the parallel version run over two processors.

Here are the times:


Run 1 298 seconds
Run 2 293 seconds
Run 3 207 seconds

There are some minor differences in the test results between running the standard and new versions of run-tests, in summary these are:


             Run 1        Runs 2&3
PASS         6151          6129
SKIP          446           447
XFAIL          28            27
WARN            0             2
FAIL           11            12
BORK            0            19

These differences are mainly accounted for by the new version of run-tests.php being much stricter in what it allows in a test case. For example, any empty section will cause a ‘BORK’, and section it doesn’t recognise will also cause a ‘BORK’. The next job on my list is to go through the tests and either fix tests (or my code) so that the old and new versions give exactly the same results.

After that I’d like to find an 8 way machine and get some better data points. Any offers?