Running ‘redirected’ PHP tests

The first step in implementing the REDIRECTTEST section in parallel runtests code is to get some redirected tests running on my Mac. I’ve used PDO and MySQL, principally because I already had MySQL installed so it seemed the easiest place to start.

This is what I had to do to get them to run:

$cd php/source/code
$./configure --with-zlib --with-pdo-mysql 
         --with-mysql-sock=/var/mysql/mysql.sock
$make

Missing the location of mysql.sock results in tests being skipped and errors like this: “SQLSTATE[HY000] [2002] No such file or directory”, it took me a while to work out what it meant.

The PDO tests assume the following if you do not specify properties using environmental variables:

  1. The MySQL database has a user called ‘root’
  2. The ‘root’ user has no password
  3. There is a database called ‘test’

The alternative to using the defaults is to assign some environment variables as follows:

$export PDO_MYSQL_TEST_DSN="mysql:host=localhost;dbname=test"
$export PDO_MYSQL_TEST_USER=your_mysql_uid
$export PDO_MYSQL_TEST_PASS=your_mysql_pwd

After setting those things up I was able to run:

$export TEST_PHP_EXECUTABLE=/php/to/test/php
$php runtests.php  ext/pdo_mysql/tests/common.phpt

The second command assumes that I’m in the top level directory of the PHP source code. The summary output is as follows:

Number of tests : 62 60
Tests skipped : 2 ( 3.2%) ——–
Tests warned : 0 ( 0.0%) ( 0.0%)
Tests failed : 0 ( 0.0%) ( 0.0%)
Expected fail : 0 ( 0.0%) ( 0.0%)
Tests passed : 60 ( 96.8%) (100.0%)

Which is good enough to start work with. I guess the next thing is to work out what the REDIRECTTEST part actually does.

 

Parallel run-tests performance

The development of parallel run-tests.php is moving along. Unfortunately paid work is going to get in the way a bit for the next few weeks so things will move a little slowly.

I’m still thinking about the REDIRECTTEST implementation. I don’t think it’s that hard, the other thing that needs to be thought through is exactly how to avoid clashes between, for example, msql and mysqli tests. I think that the latter is best addressed by having a configuration file of test directories that have to be run in sequence. In a parallel run, one processor would be set to work through these while other tests were scheduled randomly to the remaining processors. Anyway – it needs some thought and the parallel execution code is the part I’m least familiar with.

The other thing that really needs to be done is some more extensive performance work. I have done a couple of runs on a dual core Mac, just to get some data points.

The tests in the timing benchmark (phpruntests/QA/QATimedBucket.tgz) are taken from the PHP development stream and are all tests under:



ctype date dom ereg fileinfo filter iconv json libxml pcre phar 
posix reflection session spl sqlite3 standard tokenizer xml 
xmlreader xmlwriter zlib

 

I executed these tests three times. Run 1 uses the current PHP development stream version of run-tests.php. Run 2 uses the parallel version of run-tests.php run in sequential mode and Run 3 uses the parallel version run over two processors.

Here are the times:


Run 1 298 seconds
Run 2 293 seconds
Run 3 207 seconds

There are some minor differences in the test results between running the standard and new versions of run-tests, in summary these are:


             Run 1        Runs 2&3
PASS         6151          6129
SKIP          446           447
XFAIL          28            27
WARN            0             2
FAIL           11            12
BORK            0            19

These differences are mainly accounted for by the new version of run-tests.php being much stricter in what it allows in a test case. For example, any empty section will cause a ‘BORK’, and section it doesn’t recognise will also cause a ‘BORK’. The next job on my list is to go through the tests and either fix tests (or my code) so that the old and new versions give exactly the same results.

After that I’d like to find an 8 way machine and get some better data points. Any offers?

Update – parallel run-tests.php

A few more people sent me performance figures for parallel run-tests.php last night, thanks to Olivier Doucet, Stefan Marr and Sean Coates and Chris Jones we have a few good points on the graph.

A few things to note:

  1. We are mainly running different sets of tests. The biggest runs are mine and Sean’s which have 6636 tests and run as 47 groups. This set of tests can be found in ~/phpruntests/AQ/QATimedBucket.tgz
  2. I’ve removed any variation in CPU and processor type by just plotting times as a fraction of the sequential time
  3. Sean ran the 47 groups over 47 processes, the time for this was 0.28 of the single process time. It looks from these results as though distributing between 4 processors is optimal

Thanks very much for running these!

SketchUp projects

SketchUp is a funny sort of thing; it used to belong to Google but never really seem to fit with Google-anything-else. Someone once told me that it looked like a product that was conceived when the Google executives got rich and wanted something to help design their new homes. In April 2012 SketchUp was acquired by Trimble, I really hope it’s going to have a great future, I’m using the free version at the moment but it’s so useful that I probably will end up paying for it.

Over the Christmas holiday this year I built a table – using a piece of slate that we bought about 30 years ago intending to ‘do something with it’, some oak and a bit of custom made stainless steel. I used SketchUp to help me plan out the project.

This is a screen shot of the initial design:

Coffee table design

Coffee table design

The angles between the stainless steel rods and thus the placement of the legs were dictated by the dimensions of the  slate. The table looks like this upside down

Upside down view

In this view I have pulled one of the legs away slightly to show the construction of the top of the legs – this is also shown in more detail in the inset. The most useful part of SketchUp was that it helped me to work out the exact saw cuts that I needed to make – then move pieces around to that I could see what the result looked like.

Other useful things were being able to send images to Alan at Designs in Stainless who made up the stainless steel part and being able to deconstruct the wooden part of the design into a cutting layout so that I knew how much oak to order. By the way, the oak came from Alresford Interiors who make the kind of fitted kitchens that I’m going to have in my dream home.

I’ll continue using SketchUp for this kind of project – I’m sure that it probably wasn’t what it was intended for but I don’t think I could plan this sort of construction without it. Here is a shot of the finished table:

Finished table

Setting up an Android environment for PhoneGap

 

Starting to use  PhoneGap – aka Apache Cordova has been a frustrating experience, not because of PhoneGap but because of getting the Android environment set up. The getting started (Android) tutorial for PhoneGap is great. Once I had the environment set up, working through the tutorial took a few minutes. I am not going to reproduce anything that is in that tutorial here, this post is just about how to get to the point where you can work through the tutorial. I’m working on a Mac (Snow Leopard) so these instructions are Mac specific.

 

There are 4 steps:

  1. Downloads
  2. Installing the ADT Eclipse Plugin
  3. Installing the Android SDK
  4. Setting up an Android Virtual Device.

 

1. Downloads

Get started by downloading the major components.

  1. Eclipse Classic 3.7.1
  2. Android SDK R16 MacOSX (Note: This is an ‘SDK starter kit’, see later)
  3. PhoneGap 1.4.1. You don’t need this to set up the Android environment but you will need it for the PhoneGap tutorial so grabbing it while you are downloading stuff makes sense.

I extracted these into 3 separate directories under ~/Applications/ECLIPSE, ~/Applications/ANDR and ~/Applications/PHONEGAP, I don’t think it matters much where they go. Just remember the location.

 

2. Install the Android Developer Tools (ADT) plugin for Eclipse.

Start up Eclipse, I normally do this from the command line so I just executed:

./Applications/ECLIPSE/eclipse/eclipse &

With the brand new install of Eclipse running,  you can  install the ADT plugin:

  1. Help->Install New Software.
  2. Click the ‘Add’ button, top right.
  3. Type ‘ADT Plugin’ in the box labelled ‘Name’
  4. Paste this location “https://dl-ssl.google.com/android/eclipse/” into the box labelled ‘Location’.
  5. Follow the instructions. These are straightforward, so I won’t reproduce them.

 

3. Install the Android SDK

First, tell Eclipse where you put the SDK starter kit. To do this:

  1. Eclipse->Preferences->Android
  2. Paste the location of our SDK into the box and click ‘Apply’

This part is confusing because nothing happens except this:

The SDK Platform Tools component is missing!??

I spent a long time wondering what I had done wrong.  In fact this is the expected behaviour, all that this step achieves is to tell Eclipse where the ‘SDK starter kit’ can be found so that in the next step it can install the SDK. Just click ‘OK’ to get rid of the message and ‘OK’ again to close the window.

Now that Eclipse knows where you have put the ‘SDK starter kit’ you can go ahead and install whatever version of the SDK you need.

  1. Window->Android SDK manager
  2. A window appears with all SDKs you might want to install (see below), pick whatever you need and install it.

 

Android SDKs to install

Android SDKs to install

This takes a while, in the end you should see something like this:

Success installing SDK

Success!

Now, if you return to the the Android SDK pages in Eclipse, (Eclipse->preferences->Android) you will see that Eclipse knows about the SDK:

 

Android SDK is installed

The Android SDK is installed!


 

4. Setting up an Android Virtual Device (AVD)

The final piece that you will need to run the PhoneGap sample is to install an Android Virtual Device, this is a ‘phone emulator that is going to display your PhoneGap (Android)  application.

  1. Window->AVD Manager ->New
  2. Give the AVD a name (whatever you want)
  3. Under ‘Target’, select the SDK that you just installed.
  4. Give it a size. I have no idea what is appropriate here, I allocated it 10 (MiB) and things seem to work.

That’s it. If you have followed through this you have everything that you need to run the PhoneGap Android tutorial.

 

Parallel run-tests.php – development environment

In 2009 Stefan Priebsch, a GSOC student and I wrote a version of run-tests.php that runs PHPTs in parallel. This helped reduce run times on multi-core machine by a decent amount. In September 2009, IBM wanted me to work on OSGi – a fun job but completely impossible to combine with PHP development – Stefan was busy with thePHP.cc and the GSOC student took the money and ran. So development stopped – which was a pity because all we really had left to do was implement –REDIRECTTEST–

 

A couple of weeks ago I decided that I’d look to see if the code still ran. It does, but there are a few fixes needed which I am working on now, after that I’ll implement the missing sections.

 

Having not worked on PHP for a while I found that the most painful thing was recreating the development environment that I had in 2009. Here is a brief list of what I had to do in terms of set up on a Mac, it assumes that everything necessary to compile C (eq gcc, make, autoconf…) and an svn client is already installed.

  1. Get PHP from here http://www.php.net/downloads.php. ( I used the PHP 5.4.0 tar file)
  2. Extract it.
  3. Run ./configure –with-zlib –exec-prefix=/usr/local/php540
  4. make
  5. sudo make install

This gets PHP installed as /usr/local/php540/bin/.

After that I installed:

    1. PHPUnit (using pear)
    2. Phing (pear again)
    3. XDebug 2.2.0 (downloaded, built and installed).
    4. Doxygen (download tar, configure; make;  sudo make install;)

The next step was to checkout the parallel run-tests.php code,

svn co https://svn.php.net/repository/
              php/phpruntests/trunk phpruntests

cd phpruntests

Before trying to run anything there is a configuration file that needs to be set up, ‘phpdefinitions.txt’, which can be found in the phpruntests directory. Open it up and modify it as indicated to add the paths to various PHP executables.

You may wonder why we don’t just set the environmental variables, TEST_PHP_EXECUTABLE, TEST_PHP_CGI_EXECUTABLE. The answer is that because the unit tests test parts of the code that check which PHP executable to use we will ignore any pre-existing settings. When running unit tests, we specify which version of PHP to use using the -p flag.

The php versions are read from phpdefinitions.txt in tests/rtTestBootstrap.php and stored in global variable called RT_PHP_PATH and RT_PHP_CGI_PATH.

The phpdefinitions.txt file is also used as a phing properties file. The PHP executables in this case are used in the QA target.

Unit tests can be run using

phing test 

A more extensive set of tests can be run by using this command:

phing qa 

Running this will run the unit tests, generate test coverage, doxygen documentation and compare the output from a set of tests run using the new code with the same set run using the old code. It all takes a while to run – 15 minutes on my Mac.

To make life slightly easier for myself I downloaded Eclipse PDT, added Subclipse to it and added PHPUnit as an external tool. This last bit means that I can highlight unit tests and run them from Eclipse.

Next steps – work through my code and see if I can remember how it works.

 

 

 

Running PHP tests – the basics

Before diving into how the parallel code works it is worth just covering how run-tests.php works – at a very simple level.

The easiest way to run a PHP test case is from the command line, like this:

The run-tests.php code parses the file “mytest.phpt”, finds the php code in the FILE section and runs it like this:

The run-tests.php code goes on to compare the output from the above with some expected test output. If they are the same the test passes, if not it fails.

As I said, this is a very simple example. The run-tests.php code can be thought of in three parts:

  1. The code required to set up the test environment and “php_options
  2. Code that handles running tests.
  3. Retrieving, comparing and displaying output.

Over the next few posts I’ll work through each of these stages, explaining how they are handled by the new parallel run-tests code.