The development of parallel run-tests.php is moving along. Unfortunately paid work is going to get in the way a bit for the next few weeks so things will move a little slowly.
I’m still thinking about the REDIRECTTEST implementation. I don’t think it’s that hard, the other thing that needs to be thought through is exactly how to avoid clashes between, for example, msql and mysqli tests. I think that the latter is best addressed by having a configuration file of test directories that have to be run in sequence. In a parallel run, one processor would be set to work through these while other tests were scheduled randomly to the remaining processors. Anyway – it needs some thought and the parallel execution code is the part I’m least familiar with.
The other thing that really needs to be done is some more extensive performance work. I have done a couple of runs on a dual core Mac, just to get some data points.
The tests in the timing benchmark (phpruntests/QA/QATimedBucket.tgz) are taken from the PHP development stream and are all tests under:
ctype date dom ereg fileinfo filter iconv json libxml pcre phar posix reflection session spl sqlite3 standard tokenizer xml xmlreader xmlwriter zlib
I executed these tests three times. Run 1 uses the current PHP development stream version of run-tests.php. Run 2 uses the parallel version of run-tests.php run in sequential mode and Run 3 uses the parallel version run over two processors.
Here are the times:
Run 1 298 seconds Run 2 293 seconds Run 3 207 seconds
There are some minor differences in the test results between running the standard and new versions of run-tests, in summary these are:
Run 1 Runs 2&3 PASS 6151 6129 SKIP 446 447 XFAIL 28 27 WARN 0 2 FAIL 11 12 BORK 0 19
These differences are mainly accounted for by the new version of run-tests.php being much stricter in what it allows in a test case. For example, any empty section will cause a ‘BORK’, and section it doesn’t recognise will also cause a ‘BORK’. The next job on my list is to go through the tests and either fix tests (or my code) so that the old and new versions give exactly the same results.
After that I’d like to find an 8 way machine and get some better data points. Any offers?