Posted on

Selenium IDE Running Suites of Suites

Selenium Add On Banner

For over a year now I’ve been running over a dozen Selenium IDE test suites every time I update the Store Locator Plus base plugin.  It is a manual process that is time consuming, though less consuming than manual testing by several orders of magnitude.   Today I learned how to be even more efficient with my time, which my forthcoming customer support and QA team will hopefully appreciate when they come on-board this summer.

Here is the Stack Overflow summary I posted on my automated “suite of suites” process which took me several days of searching and testing to discover.

I have a few dozen test suites built in Selenium IDE to assist with testing my Store Locator Plus WordPress plugin. Sometimes I need to run a single Selenium test suite. However when I release a new version of the base plugin I want to run a dozen test suites one-after-another.

While not a perfect fit for your use case of creating several “master suites”, I did find a pair of Selenium IDE plugins that allow me to create a single “favorites list of suites” and run all of my favorites back-to-back.

It may be possible to investigate & modify the plugin JavaScript to create several different “favorites lists” that may suit your needs. In the meantime you can get at least one “master list of suites” by combining these Selenium IDE add-ons:

After installing each of these add-ons (technically Mozilla Firefox plugins) you will see a favorites button inside the Selenium IDE interface. Mark your favorite suites and you will have your “list”. You can now select “Favorites / Run All” from the Selenium IDE menu.

You may want to be careful about the sequence in which you mark your favorites. I marked them in the order I wanted them to run. Open test suite #1, favorite, test suite #2 favorite etc. then “run all”. Worked great and shows me the total run count and fail count across all suites (and thus tests) that were executed. The log, sadly, appears to be reset at each suite however.

Posted on

Is phpUnit Good Enough For Testing WordPress?

I’ve been using Selenium IDE to test my Store Locator Plus plugin for the past year.   At nearly every WordCamp I attend I hear about using phpUnit to test “WordPress things”.   I heard about it at WordCamp Orlando during the contributor session; one of the first things Mark Jaquith asked me about a proposed patch to dbDelta was “is there a unit test for dbDelta?”.   If you Google “testing WordPress plugins” it is all about unit tests.

phpUnit is clearly the defacto standard for testing WordPress code.

My First Plugin phpUnit Test

Nine months after talking to Bryan Petty about phpUnit testing at WordCamp Atlanta, I finally got around to adding phpUnit tests to my plugin testing arsenal.   Since Mark provided a few hints on getting WordPress Core code pulled into my development environment I had the necessary tools to get started.    I created a new testing folder in my plugin assets directory, threw in the bootstrap, phpUnit XML directives config file, and added my first unit test class.

<?php

/**
* phpUnit Test for SLPlus
*
* @package PHPUnit\StoreLocatorPlus
* @author Lance Cleveland <lance@charlestonsw.com>
* @copyright 2014 Charleston Software Associates, LLC
*/
class SLPlusTest extends PHPUnit_Framework_TestCase {

/**
* @var \SLPlus
*/
public $slplus;

/**
* @var int
*/
public $location_id;

/**
* Setup the global pointer to the main SLPlus plugin object.
*/
function setUp() {
parent::setUp();
$this->slplus = $GLOBALS[‘slplus_plugin’];
}

/**
* Printout before this test suite runs.
*/
public static function setUpBeforeClass() {
print "\n".get_class()."\n";
}

/**
* To next line before next test suite runs.
*/
public static function tearDownAfterClass() {
print "\n";
}

/**
* Is the slplus variable an SLPlus object?
*/
function test_slplus_is_set() {
$this->assertInstanceOf( ‘SLPlus’, $this->slplus );
}
}

Fairly simplistic test. It told me that I was on the right path to getting phpUnit tests working with my plugins.

Putting The Test Into Production

I added more tests with more complexity and eventually added phpUnit tests to my WordPress Development Kit production system, a Grunt-based series of scripts I use on my WordPress Development vagrant boxes.

Nice! I now have a low-level code-based system for testing functionality without having to run everything through an IDE. Selenium IDE is great for testing but it is a lot slower for a coder to write test scripts and the logic is far more restrictive.

This is going to be great!

phpUnit Imperfection

Within a few days I found the first chink in the phpUnit armor.

In my Selenium IDE tests I have a series of tests that check to see if PHP warnings like “Notice: Undefined…” , “Fatal error:…” , and other calamities pop up in the browser while running the tests.   Since I am always running with “DEBUG mode full on” on my development boxes, these errors pop up any time I fat-finger the code or have a brain-fart and forget to quote an array index.

While refining my Selenium IDE scripts, by adding rollups to increase consistency among the tests, I purposefully added a common coding error by typing asdf at the top of my base plugin class.    I re-run my Selenium IDE scripts.  They catch the error, exactly as they are supposed to.

Selenium Catches PHP Notices
With the right test command, Selenium makes PHP warnings and notices obvious.

I start wondering… “I wonder what phpUnit will do, I bet it is much faster at catching this sort of thing”.

NOPE.

phpUnit spit out a warning at the very top of the test, which quickly scrolled out of view.   At the end of a long test series I see all my tests passed.    If I didn’t scroll back to the start of the test series I’d never have seen the warnings and notices that were buried in the startup output from the phpUnit testing.

phpUnit Notices Not Errors
phpUnit does not make notices blatantly obvious.

For my personal workflow, where I am constantly multitasking, having those PHP warnings and notices force a proper phpUnit test ERROR would be a far better option.   In “go mode” I am only going to look at the final summary line.   OK (5 tests, 14 assertions)  DONE, WITHOUT ERRORS. If there is a notice or warning I want to see something like “DONE, WITHOUT ERRORS. LOTS OF NOTICES. LOTS AND LOTS OF NOTICES. FIX THEM!!!!”. Maybe not exactly like that, but you get the idea.

True, these were not errors, but a blatant PHP notice spitting out the the screen is not within my code standards.  Yes, it goes away if you do not run in debug mode, but in my world plugin code should not be generating errors, warnings, OR NOTICES.   Not only do they indicate a deeper issue with possible unforeseen ramifications but it often fills up log files, which means writing to disk, which directly affects performance of the WordPress system as a whole.

Doing It Wrong?

Since I am new to phpUnit I may be doing it wrong.

My phpunit.xml configuration:

<phpunit
        bootstrap="bootstrap.php"
        backupGlobals="false"
        colors="true"
        convertErrorsToExceptions="true"
        convertNoticesToExceptions="true"
        convertWarningsToExceptions="true"
        >
    <testsuites>
        <testsuite>
            <directory prefix="" suffix="Test.php">./</directory>
        </testsuite>
    </testsuites>
</phpunit>

I even tried this trick to force errors thanks to a hint from Bryan.

The default phpUnit setup and the plugin testing examples online indicate I have the correct setup in place.   My guess is other “phpUnit first timers” will end up with a similar configuration.  A configuration that does a less-than-stellar job of catching PHP warnings and notices.

Selenium IDE Required

Selenium IDE, on the other hand, makes PHP warnings, notices, and errors blatantly obvious if you test for them.   If you are running with debug logging enabled, which you ALWAYS ARE on your dev box because you are a coder that gives a damn, Selenium will flag the output when it is rendered on your WordPress pages.   Selenium highlights the errors in red and reports them in the test summary as failures.   This is something I see even in “go mode”.

Bottom line, I feel as though you must have both phpUnit and Selenium IDE (or Selenium Server) to properly test a WordPress code implementation of any kind.   Not too mention it is nearly impossible, as far as I know, to test advanced JavaScript and AJAX functionality with phpUnit.

Now I am curious:  Does WordPress Core have anything in place to perform similar user-interface testing?  Are they running anything like Selenium IDE to ensure the interface renders as expected?  The CSS is correct?  AJAX calls are running and producing front-end results?   Can phpUnit catch front-end output like errant “Notice: use  of undefined constant” messages when running in debug mode?

I’m sure some of you know a LOT more about this stuff than I do.   Share you knowledge in the comments!

If you want to know more about my Selenium IDE scripts, Grunt production helpers, and Vagrant boxes ping me here as well.   Everything I do is open source and freely available to the WordPress development community.

Posted on

Selenium IDE Includes AKA “Rollups”

Selenium IDE is used extensively to test the Store Locator Plus family of plugins.    As the testing continue to expand so do the rules being tested.   Today the addition of a rule that checks that the PHP notice “use of undefined constant” needed to be added.     There is an existing set of other PHP warnings and notices that was added last month that is in 30+ Selenium Scripts.  They were all copied by hand into each test.

Now I need to add one more rule to that test set.   Copy and paste into 30 files?  There HAS to be a better way.

Turns out there is a better way, but it requires a little JavaScript coding to make it happen.   Selenium IDE does not have an “include <file>” option in the base set of commands.   Personally I think they need to add it to the base command set as it will make it far easier for people to write “test subsets” and then include them in every test.    The solution is using something called a “rollup”.    A rollup is a label for a group of commands you want to execute in many places in your test scripts.

My original test suites looked something like this:

Selenium Test for PHP Warnings
Selenium Test for PHP Warnings

In the “old days”, yesterday to be precise, I would copy-and-paste this set of Selenium IDE tests into a LOT of test files.

Today I decided to be smart about it, happens every now-and-then… must be a full moon or something, and find out how to to an “include” of those commands.   The trick is to create a file called something like “rollups.js”.     My file is called slp_rollups.js.     It is a standard JavaScript file that I place in the same directory as all of my Selenium IDE test scripts (which happen to be nothing more than HTML snippets).

To replace those 4 commands with a rollup I created this slp_rollup.js file:

/**
 * For use in Selenium IDE.
 *
 * You will need to add this to Selenium under Options / Options in the Selenium Menu.
 * Put this under Selenium Core Extensions:
 * ~/selenium-ide-for-slp/sideflow.js , ~/selenium-ide-for-slp/slp_rollups.js
 *
 *
 */
var manager = new RollupManager();


/**
 * check_for_syntax_errors
 *
 * This rollup tests for php syntax errors, warnings, and notices.
 */
manager.addRollupRule({
    name: 'check_for_syntax_errors',
    description: 'Check for PHP syntax errors, notices, warnings.',
    args: [],
    commandMatchers: [],
    getExpandedCommands: function(args) {
        var commands = [];

        commands.push({
            command: 'assertNotText',
            target: '//body',
            value: '*Notice: Undefined*'
        });

        commands.push({
            command: 'assertNotText',
            target: '//body',
            value: '**Notice: Trying to get*'
        });

        commands.push({
            command: 'assertNotText',
            target: '//body',
            value: '*Notice: Use of*'
        });

        commands.push({
            command: 'assertNotText',
            target: '//body',
            value: '*Fatal error:*'
        });

        return commands;
    }
});

To activate this I update Selenium IDE by going to Options/Options and adding this new slp_rollup.js file to the Selenium Core Extensions. Since I also use the sideflow.js file to provide Go To / If and other looping constructs I add both sideflow.js and slp_rollups.js to my extensions list by separating the file names with a comma.

Selenium IDE Options
Selenium IDE Options with sideflow and slp_rollups enabled.

Now I can replace that block of 4 commands with the following single command in ALL 30 scripts. The best part is the next time I need to add another test for a new warning or error I only edit ONE file, the slp_rollups.js file which means less editing, less copy & paste, and less commits to the git repository.

Selenium IDE Implementing Rollups
Selenium IDE Implementing Rollups
Posted on

Adding gotoIf and other Flow Control to Selenium IDE

selenium ide

Some of my Selenium IDE test scripts make use of the gotoIf command.   Flow control logic like the gotoIf and label statements are not part of the standard Selenium IDE core library.   Like most apps these days, Selenium IDE has a method for extending the base functionality with plugins and core extensions.   The “sideflow” core extension from darenderidder is an extension that provides the oft-referenced goto controls for Selenium IDE.

Adding sideflow Flow Control To Selenium

I like to keep current with any repository updates, so I use git to clone the repository into my documents folder on my Vagrant WP Dev Kit CentOS box.    Using SmartGit I clone via the github URL:

https://github.com/darrenderidder/sideflow.git

Open Selnium IDE, go to the Options menu and select options.  Under the Selenum Core extensions browse to the sideflow.js file that was cloned via git.

Installing Selenium IDE SideFlow Extension
Installing Selenium IDE SideFlow Extension

The other option is to just download the sideflow.js file here.   No promises or guarantees this is current or will work on every system.  You should really grab the code from the source.

Example Selenium IDE Script

Here is an example script that uses flow control to ensure only ONE Store Locator Plus locations page is added. If the page already exists it skips the page setup process on my test site.

<tr><td rowspan="1" colspan="3">SLP - Add Page, SLPLUS</td></tr>
</thead><tbody>
<tr>
	<td>setSpeed</td>
	<td>200</td>
	<td></td>
</tr>
<!--Open WP Pages Interface-->
<tr>
	<td>open</td>
	<td>/wp-admin/edit.php?post_type=page</td>
	<td></td>
</tr>
<tr>
	<td>waitForElementPresent</td>
	<td>id=doaction2</td>
	<td></td>
</tr>
<tr>
	<td>storeElementPresent</td>
	<td>xpath=//table[@class='wp-list-table widefat fixed pages']//td[@class='post-title page-title column-title']//a[contains(text(),'Locations')]</td>
	<td>slplus_page_exists</td>
</tr>
<tr>
	<td>gotoIf</td>
	<td>storedVars['slplus_page_exists']</td>
	<td>SKIP_LOCATION_PAGE_CREATION</td>
</tr>
<tr>
	<td>clickAndWait</td>
	<td>css=a.add-new-h2</td>
	<td></td>
</tr>
<tr>
	<td>type</td>
	<td>id=title</td>
	<td>Locations</td>
</tr>
<tr>
	<td>click</td>
	<td>id=content-html</td>
	<td></td>
</tr>
<tr>
	<td>type</td>
	<td>id=content</td>
	<td>[[slplus]]</td>
</tr>
<tr>
	<td>waitForElementPresent</td>
	<td>id=publish</td>
	<td></td>
</tr>
<tr>
	<td>clickAndWait</td>
	<td>id=publish</td>
	<td></td>
</tr>
<tr>
	<td>label</td>
	<td>SKIP_LOCATION_PAGE_CREATION</td>
	<td></td>
</tr>
Posted on

Simple Web App Testing

After all the discussions about testing web apps and working with a variety of relatively horrible app testing platforms, I have been learning Selenium IDE.  Selenium IDE is the “little brother” of the full blown Selenium Web Driver.    Where Web Driver is a robust client/server framework with distributed testing and full test logic options, IDE is a simple light weight scripting plugin for your browser.

 That is what I like about IDE.   I know what it is and don’t push it beyond those limits.  It is basically a “web input” record & playback utility with a few commands added to make it a rudimentary testing system. I now use it for testing the WordPress plugin base functionality with limited effort.   Spending an extra 10 minutes writing a test case pays off in numerous “one-click-and-done” script runs in the future when I want to do something like “make sure the Pro Pack name search is not broken” after a new release.
The premise is simple, when you add Selenium IDE to your browser as a plugin you get a menu option in the browser “Selenium IDE”.   You can record a script by clicking a record button, “do you web thing”, then stop recording.  You now have a script outline.   Click the “run this button” on the interface and the script runs again.
One caveat.   The recorded scripts are not very good.  They get the job done for a quick start, but you are much better off spending and extra 5 minutes and writing the test commands by hand. They are far more efficient and accurate.

Getting Started

When you select it a new popup window opens with a simple test case table on the left and commands table on the right.   The Test Case table is mostly for Test Suites (groups of test cases you run in sequence) so we’ll ignore it for now.    On the right side is the commands and it start out with a “blank slate” of commands.   I have another blog post about the commands I use that handle 90% of my test scripts that you can use as a quick cheat-sheet.
To start a new script you use the right-side command window.  The general command sequence is to  click on a new blank line, start typing a command in the bottom-of-screen command box and press enter to auto-complete, enter a target which identifies an on-screen element in jQuery-like fashion, and a value for certain commands.  Repeat.

An Example Session

Here is how I create a simple test to make sure SLP Pro Pack is installed and allowing lat/long editing to occur:
  • Click the first blank line.
  • Select command on the bottom entry form.
  • Start typing “open”.  This is the “open  a web page in same window” command.
  • Select target.  type the relative path (you can use a full http address, but I want my scripts to run on any server)
  • Click on the next blank line & continue building out my script.

I won’t bore you with each command, here is a screen shot of the Pro Pack Edit Location lat/long script that is also on GitHub.

selenium script example
selenium script example

Commands

The commands are detailed in my other post, but the quick summary:

  • open = open a web page
  • waitForElementPresent = pause execution until something that needs to be on the page is present
  • clickAndWait = click an element and wait for the page to reload before continuing
  • type = type text into an HTML element, like an input box
  • assertValue = flag an error and STOP SCRIPT EXECUTION if the value does not match

Targets

The “target” element of the script commands can be tricky for people that do not know XHTML or JavaScript/jQuery style targets.   I find the easiest way to manage this is to make sure my web apps have discrete input names or IDs on most elements or useful and distinct class names for IDs that may be dynamic like “location-id-33”.   With a unique name or ID you can get to your target element easily, just type name=foo or id=bar and you are there if it is the only element with that ID or name.

For items without a discrete name or with multiple instances you can combine XHTML-like references with other elements.   Two key things to remember about this, using a single-slash means “exactly at this point in the document hierarchy and double-slashes means “first found”.     For example /table/thead/tbody/tr[1] is THE first row after the table, thead, tbody elements.  If any are missing it will fail.    In contrast /table//tr[1] will meant the first table row that can be located after the first table in the document.    I often mix-and-match “select by name” and the double-slash “first found” options so that something like this shows up:

//table[@id=’foo’]//tr[2]//th[2]//a[@class=’bar’]

This is useful in WordPress where I can never be certain that a theme will not change the page structure.  In this case it says “find the first table element with an id set to foo, within that find the second tr element (since a theme/plugin may throw in some thead or other element I continue the // format), find the second th element in that row, and select the first hyperlink with a class set to ‘bar’.

Targets take some getting used to, but luckily you can test your “guess” at the right path after it is entered with the “find” button that will go show you which element it thinks you mean on the page that is currently rendered in the browser window.

What I’ve Learned

A couple of important things I’ve learned along the way.   First, use waitForElementPresent judiciously.   I usually put this in place before I interact with the first item on the screen.  The “…AndWait” commands wait for the page to load but with so much jQuery page loaders out there it is common for the element you need to not be ready when the page first comes online.   The “waitFor…” commands can wait a long time and will not always flag an error if they time out.    Use “assert…” whenever something critical must be on the page or to flag a script as failing.   “assert…” is NOT the only way a script will fail, which is a feature I like about Selenium IDE.  If you try to click an element and it is not there it fails the script but tries to continue running.  This makes it very easy to write light-weight test scripts.

No Excuses

Overall it is not as robust as many testing platforms, but that is a good thing IMO.   Because it is light & easy you can get Selenium IDE in place and get your first test scripts online in well under an hour.    That means you really have no excuse not to be doing at least RUDIMENTARY repetitive task testing on your web apps.

I’ve started using this to test my WordPress plugins and try to add at least one new short test case in each plugin release.  It adds about 10-15 minutes on average to the test cycle but it means one more feature is tested before each release.  Even now with just a dozen test suites I’ve prevented at least two significant functional bugs from getting out to the public.

Use Selenium IDE and do some web tests.   The user community will thank you for it.

### Technorati Claim Code CGBQWRFD3GA5 ###
Posted on

Selenium IDE Commands

I’ve been using Selenium IDE for a few months now with very simple test scripts.   Mostly scripts I’ve recorded to test my WordPress plugins.   They are not very sophisticated, but as my release schedule has increased so has my need for more robust testing.  Unfortunately the Selenium IDE 2 documentation is sparse at best.   Maybe they have a complete list of commands and how they work, but I can’t find it.

So I decided to create my own Selenium IDE commands list.   This is my cheat sheet and is based on my current knowledge as an end user.  It may not be accurate, so always check what you read here before applying it to a production test suite.

Using Selenium IDE

One of the first lessons I learned is that recording/playing back scripts is a great way to get started.   The second lesson learned seconds later is that the scripts are far from perfect, very inefficient, and always need editing.

Add wait commands, it is key to successful tests.   Always make sure an element is loaded before running full speed through the script.

Edit the first couple of commands to go directly to the test URL.  Often the scripts start elsewhere.

Realize this is not a full programming language.  Out of the box it does NOT support if, goto, or loop constructs.  It is a pure linear command execution.  If you need more you should get an add-on OR use the full Selenium RC line.

Selenium IDE Commands

(that I use)

assertElementPresent

Arguments:

locator – an element locator

Description:

Make sure an element is present, if not fail and abort the remainder of the test.

Notes:

I use this to make sure that a previous test suite or action did what I expected.  If not, something is horribly wrong.   Assert commands tell Selenium IDE “hey, something bad happened… stop testing”.

assertElementPresent

Arguments:

target – a piece of javascript that returns a value

value – what you expect the value to be set to

Description:

Make sure the javascript executes.  The find button will not work here, you must execute the command to test this.

Notes:

This is a recent addition to my Selenium IDE arsenal.  I use it to check custom CSS rules and make sure that when I tweak a CSS rule via my admin panel textarea field that it makes it to the front end.   I check the computed style of a specific element and make sure the attribute is set as I expect.  For example, check that the final style sets a font size to 48px.

Example target:

window.document.defaultView.getComputedStyle(window.document.getElementsByClassName(‘entry-title’)[0]).getPropertyValue(‘font-size’)

Example value:

48px

check

Arguments:

locator – an element locator

Description:

Check a toggle-button (checkbox/radio).

Notes:

This is a great way to force a checkbox on.   The mirror command is uncheck.

  • command: check
  • target: you element finder i.e. name=checkbox_name_property
  • value: n/a (leave blank)

click

Arguments:

locator – an element locator

Description:

Clicks on a link, button, checkbox or radio button. If the click action causes a new page to load (like a link usually does), call waitForPageToLoad.

Notes:

A good idea to use waitForElement to make sure your locator is present before doing this.

open

Arguments:

url – the URL to open; may be relative or absolute

Description:

Opens an URL in the test frame. This accepts both relative and absolute URLs. The “open” command waits for the page to load before proceeding, ie. the “AndWait” suffix is implicit. Note: The URL must be on the same domain as the runner HTML due to security restrictions in the browser (Same Origin Policy). If you need to open an URL on another domain, use the Selenium Server to start a new browser session on that domain.

select

Arguments:

selectLocator – an element locator identifying a drop-down menu
optionLocator – an option locator (a label by default)

Description:

Select an option from a drop-down using an option locator.

Notes:

The easiest way to select an element on a pulldown list is:

    • command: select
    • target: id=<input id>
    • value: label=<text string of item to select>

Here are notes on the selectors from Selenium IDE:

Option locators provide different ways of specifying options of an HTML Select element (e.g. for selecting a specific option, or for asserting that the selected option satisfies a specification). There are several forms of Select Option Locator.

    • label=labelPattern: matches options based on their labels, i.e. the visible text. (This is the default.)
    • label=regexp:^[Oo]ther
    • value=valuePattern: matches options based on their values.
    • value=other
    • id=id: matches options based on their ids.
    • id=option1
    • index=index: matches an option based on its index (offset from zero).
    • index=2

If no option locator prefix is provided, the default behaviour is to match on label.

type

Arguments:

locator – an element locator

value – the value to type

Description:

Can also be used to set the value of combo boxes, check boxes, etc. In these cases, value should be the value of the option selected, not the visible text.

Notes:

This is what I use to “type” data into an input field.

uncheck

Arguments:

locator – an element locator

Description:

Uncheck a toggle-button (checkbox/radio).

Notes:

This is a great way to force a checkbox off.   The mirror command is check.

  • command: uncheck
  • target: you element finder i.e. name=checkbox_name_property
  • value: n/a (leave blank)

waitForElementPresent

Arguments:

locator – an element locator

Returns:

true if the element is present, false otherwise

Description:

Verifies that the specified element is somewhere on the page.

Notes:

I try to use this before I click on anything, enter input values, etc.  Often an element is not loaded yet.  This waits until it is (or fails).

Checking if a hyperlink that contains specific text is a good way to manage the AJAX operations of WordPress categories.    You can use the following command set:

command: waitForElementPresent
Target: xpath=//a[text()='Category Name']

My Script(s)

I  have my Store Locator Plus testing scripts on a public GitHub repo for anyone that wants to see them.

Enjoy & please share your Selenium IDE wisdom here.  We can all use help when it comes to using this tool.