Posted on

Is phpUnit Good Enough For Testing WordPress?

I’ve been using Selenium IDE to test my Store Locator Plus plugin for the past year.   At nearly every WordCamp I attend I hear about using phpUnit to test “WordPress things”.   I heard about it at WordCamp Orlando during the contributor session; one of the first things Mark Jaquith asked me about a proposed patch to dbDelta was “is there a unit test for dbDelta?”.   If you Google “testing WordPress plugins” it is all about unit tests.

phpUnit is clearly the defacto standard for testing WordPress code.

My First Plugin phpUnit Test

Nine months after talking to Bryan Petty about phpUnit testing at WordCamp Atlanta, I finally got around to adding phpUnit tests to my plugin testing arsenal.   Since Mark provided a few hints on getting WordPress Core code pulled into my development environment I had the necessary tools to get started.    I created a new testing folder in my plugin assets directory, threw in the bootstrap, phpUnit XML directives config file, and added my first unit test class.

<?php

/**
* phpUnit Test for SLPlus
*
* @package PHPUnit\StoreLocatorPlus
* @author Lance Cleveland <lance@charlestonsw.com>
* @copyright 2014 Charleston Software Associates, LLC
*/
class SLPlusTest extends PHPUnit_Framework_TestCase {

/**
* @var \SLPlus
*/
public $slplus;

/**
* @var int
*/
public $location_id;

/**
* Setup the global pointer to the main SLPlus plugin object.
*/
function setUp() {
parent::setUp();
$this->slplus = $GLOBALS[‘slplus_plugin’];
}

/**
* Printout before this test suite runs.
*/
public static function setUpBeforeClass() {
print "\n".get_class()."\n";
}

/**
* To next line before next test suite runs.
*/
public static function tearDownAfterClass() {
print "\n";
}

/**
* Is the slplus variable an SLPlus object?
*/
function test_slplus_is_set() {
$this->assertInstanceOf( ‘SLPlus’, $this->slplus );
}
}

Fairly simplistic test. It told me that I was on the right path to getting phpUnit tests working with my plugins.

Putting The Test Into Production

I added more tests with more complexity and eventually added phpUnit tests to my WordPress Development Kit production system, a Grunt-based series of scripts I use on my WordPress Development vagrant boxes.

Nice! I now have a low-level code-based system for testing functionality without having to run everything through an IDE. Selenium IDE is great for testing but it is a lot slower for a coder to write test scripts and the logic is far more restrictive.

This is going to be great!

phpUnit Imperfection

Within a few days I found the first chink in the phpUnit armor.

In my Selenium IDE tests I have a series of tests that check to see if PHP warnings like “Notice: Undefined…” , “Fatal error:…” , and other calamities pop up in the browser while running the tests.   Since I am always running with “DEBUG mode full on” on my development boxes, these errors pop up any time I fat-finger the code or have a brain-fart and forget to quote an array index.

While refining my Selenium IDE scripts, by adding rollups to increase consistency among the tests, I purposefully added a common coding error by typing asdf at the top of my base plugin class.    I re-run my Selenium IDE scripts.  They catch the error, exactly as they are supposed to.

Selenium Catches PHP Notices
With the right test command, Selenium makes PHP warnings and notices obvious.

I start wondering… “I wonder what phpUnit will do, I bet it is much faster at catching this sort of thing”.

NOPE.

phpUnit spit out a warning at the very top of the test, which quickly scrolled out of view.   At the end of a long test series I see all my tests passed.    If I didn’t scroll back to the start of the test series I’d never have seen the warnings and notices that were buried in the startup output from the phpUnit testing.

phpUnit Notices Not Errors
phpUnit does not make notices blatantly obvious.

For my personal workflow, where I am constantly multitasking, having those PHP warnings and notices force a proper phpUnit test ERROR would be a far better option.   In “go mode” I am only going to look at the final summary line.   OK (5 tests, 14 assertions)  DONE, WITHOUT ERRORS. If there is a notice or warning I want to see something like “DONE, WITHOUT ERRORS. LOTS OF NOTICES. LOTS AND LOTS OF NOTICES. FIX THEM!!!!”. Maybe not exactly like that, but you get the idea.

True, these were not errors, but a blatant PHP notice spitting out the the screen is not within my code standards.  Yes, it goes away if you do not run in debug mode, but in my world plugin code should not be generating errors, warnings, OR NOTICES.   Not only do they indicate a deeper issue with possible unforeseen ramifications but it often fills up log files, which means writing to disk, which directly affects performance of the WordPress system as a whole.

Doing It Wrong?

Since I am new to phpUnit I may be doing it wrong.

My phpunit.xml configuration:

<phpunit
        bootstrap="bootstrap.php"
        backupGlobals="false"
        colors="true"
        convertErrorsToExceptions="true"
        convertNoticesToExceptions="true"
        convertWarningsToExceptions="true"
        >
    <testsuites>
        <testsuite>
            <directory prefix="" suffix="Test.php">./</directory>
        </testsuite>
    </testsuites>
</phpunit>

I even tried this trick to force errors thanks to a hint from Bryan.

The default phpUnit setup and the plugin testing examples online indicate I have the correct setup in place.   My guess is other “phpUnit first timers” will end up with a similar configuration.  A configuration that does a less-than-stellar job of catching PHP warnings and notices.

Selenium IDE Required

Selenium IDE, on the other hand, makes PHP warnings, notices, and errors blatantly obvious if you test for them.   If you are running with debug logging enabled, which you ALWAYS ARE on your dev box because you are a coder that gives a damn, Selenium will flag the output when it is rendered on your WordPress pages.   Selenium highlights the errors in red and reports them in the test summary as failures.   This is something I see even in “go mode”.

Bottom line, I feel as though you must have both phpUnit and Selenium IDE (or Selenium Server) to properly test a WordPress code implementation of any kind.   Not too mention it is nearly impossible, as far as I know, to test advanced JavaScript and AJAX functionality with phpUnit.

Now I am curious:  Does WordPress Core have anything in place to perform similar user-interface testing?  Are they running anything like Selenium IDE to ensure the interface renders as expected?  The CSS is correct?  AJAX calls are running and producing front-end results?   Can phpUnit catch front-end output like errant “Notice: use  of undefined constant” messages when running in debug mode?

I’m sure some of you know a LOT more about this stuff than I do.   Share you knowledge in the comments!

If you want to know more about my Selenium IDE scripts, Grunt production helpers, and Vagrant boxes ping me here as well.   Everything I do is open source and freely available to the WordPress development community.